HTML5 Audio tag on Safari has a delay

I'm trying to accomplish a simple doodle-like behaviour, where a mp3/ogg sound rings on click, using the html tag. It is supposed to work under Firefox, Safari and Safari iPad is very desireable.

I've tried many approaches and have come down to this:

HTML

    <span id="play-blue-note" class="play blue" ></span>
    <span id="play-green-note" class="play green" ></span>


    <audio id="blue-note" style="display:none" controls preload="auto" autobuffer> 
        <source src="blue.mp3" />
        <source src="blue.ogg" />
        <!-- now include flash fall back -->
    </audio>

    <audio id="green-note" style="display:none" controls preload="auto" autobuffer> 
        <source src="green.mp3" />
        <source src="green.ogg" />
    </audio>

JS

function addSource(elem, path) {
    $('<source>').attr('src', path).appendTo(elem);
}

$(document).ready(function() {


    $('body').delegate('.play', 'click touchstart', function() {
        var clicked = $(this).attr('id').split('-')[1];

        $('#' + clicked + '-note').get(0).play();



    });

});  

This seems to work great under Firefox but Safari seems to have a delay whenever you click, even when you click several times and the audio file has loaded. On Safari on iPad it behaves almost unpredictably.

Also, Safari's performance seems to improve when I test locally, I'm guessing Safari is downloading the file each time. Is this possible? How can I avoid this? Thanks!

I just answered another iOS/<audio> question a few minutes ago. Seems to apply here as well:

Preloading <audio> and <video> on iOS devices is disabled to save bandwidth.

In Safari on iOS (for all devices, including iPad), where the user may be on a cellular network and be charged per data unit, preload and autoplay are disabled. No data is loaded until the user initiates it.

Source: Safari Developer Library