App_offline.htm, CSS, images, and aspnet_isapi.dll

So, the site I'm working on is using urlrewriting in coordination with aspnet_isapi.dll (everything is mapped to it). I put up my app_offline.htm file, and all the text shows, however, the CSS or images aren't being served. I'm guessing they're being processed by ASP.NET due to the wildcard mapping instead of IIS. Is this correct? If so, how can I allow IIS to serve these files? Furthermore, an issue I can see arising..in the web.config for the rewriter settings:

<rewrite url="^~/images/network/(.*)/(.*).jpg$" to="~/services/ImageHandler.ashx?type=$1&amp;id=$2"/>
    <rewrite url="^~/image/view/(.*).jpg$" to="~/ServePRView.aspx?id=$1"/>
    <rewrite url="^~/asset/view/(.*).jpg$" to="~/services/ImageHandler.ashx?id=$1&amp;type=asset"/>

Thanks for the help all, -Steve

It's because if the webserver sees an App_Offline.htm file, it serves that file for every request in the site - even images. You have to serve images from another site. Or you can try something like this http://en.wikipedia.org/wiki/Data_URI_scheme