I know that git notes can be fetched after cloning using:
git fetch origin refs/notes/*:refs/notes/*
or even be setup in git config to be always fetched.
However at clone time I do not get the notes, so I have to clone and then fetch. Although I do see that using --mirror when cloning does get notes too. However my optimal setup would be that I could clone any repository without doing a mirror (since it implies --bare and would also get other unwanted refs) and get the notes too.
Is there any way to setup for example git config to fetch specific additional refs at clone time ?
The short answer is "no": at clone time, you have your choice of either cloning with the default refspec (+refs/heads/*:refs/remotes/$remote/*, where $remote is replaced with origin or the remote name you select) or with the --mirror fetch-mirror refspec (+refs/*:refs/*). As you note, --mirror implies --bare, and is probably not desirable here.
It would be easy enough to write a shell script that does a clone, then does git config --add remote.origin.fetch "+refs/notes/*:refs/notes/*", then runs git fetch, so that you need not run three commands yourself. (Whether you want the + here, and/or to rename their notes to some other reference name, is up to you. For the FreeBSD repository, I use fetch = +refs/notes/*:refs/notes/origin/* and set notesRef = refs/notes/origin/commits, a practice I copied from elsewhere without thinking about it too much—and so far I have had no reason to rethink or change it.)