TortoiseGit with openssh key not authenticating using ssh-agent

I'm setting up a git environment on Windows XP (msysGit 1.7.11, TortoiseGit 1.7.14) and trying to achieve following points :

• ssh connection on a port different than default 22
• ssh authentification handled by ssh-agent

So I create a ~/.ssh/config file :

Host gitbox
   User gitolite
   Hostname XX.XX.XX.XX
   Port 154
   PreferredAuthentications publickey
   IdentityFile "/c/Documents and Settings/kraymer/.ssh/id_rsa"

When using the git bash CLI, everything works as intended.

I'm struggling with TortoiseGit.
I first installed TortoiseGit with Plink and using Pageant to load ssh private key. The automatic authentication (Pageant) worked but setup was a fail as TortoiseGit don't recognize git repos url formatted as gitolite@gitbox/repo.git.

I then decided to install TortoiseGit using openssh client, so the config file can be read by the ssh client, and to mimic the git CLI setup.
I picked the ssh.exe shipped with msys git as ssh client in TortoiseGit settings.
When doing a git pull, the remote url is now resolved but the passphrase password is asked while I expect ssh-agent automatic authentication to occur.

Is it possible to make TortoiseGit work with ssh-agent ?
Or make TortoiseGit (Plink) aware of .ssh/config ?

Edit #1

Following @VonC advice I configured my $HOME variable.
When I click Show environment variables in TortoiseGit I now have :

 HOME=C:\Documents and Settings\kraymer  
 HOMEDRIVE=C: 
 HOMEPATH=\Documents and Settings\kraymer 

But git pull still require I enter passphrase.

No tweaks needed.

Just make TortoiseGit point to the same ssh client used by git itself, see the screenshot:

This should be C:\Program Files\Git\usr\bin\ssh.exe in latest version of Git as mentioned by Aleksey Kontsevich in the comments.