users are asked for password while using gitolite

I have successfully created gitolite-admin.git repo on server (say) 10.107.105.13. I can clone this repo on my local machine (say) 10.14.42.7 by issuing git clone gitolite@10.107.105.13:gitolite-admin. I had to add some lines in .ssh/config file to make sure that correct private key is used.

Then I have added a user dilawar to conf/gitolite.conf file and a appropriate key dilawar.pub to keys folder. I have added and commited this commit to the gitolite-admin repo. I have also added one more entry in .ssh/conf file so that a correct private key is used. But when I try to do git clone dilawar@10.107.105.13:testing, gitolite asks for the password. I am under the impression that I do not have to create user dilawar on 10.107.105.13. I have checked by logging into server that repository testing.git exists as well public-key dilawar.pub has been added to .ssh/authorized_keys.

I have also tried ssh -vvvv dilawar@10.107.105.13 to check if the correct file is being offered. Here is my .ssh/conf file.

HostName 10.107.105.13 
    User gitolite
    IdentityFile ~/.ssh/gitolite

Host 10.107.105.13
    HostName 10.107.105.13 
    User dilawar 
    IdentityFile ~/.ssh/id_rsa

What I am doing wrong?

In your config file, I see:

User dilawar

That is wrong. ssh communication to a gitolite server are always done with the same account (here gitolite).
What changes is the private key used, which will help gitolite determine your identity.

What you ~/.ssh/config file should look like is:

Host admin
    HostName 10.107.105.13 
    User gitolite
    IdentityFile ~/.ssh/gitolite

Host dilawar
    HostName 10.107.105.13 
    User gitolite
    IdentityFile ~/.ssh/id_rsa

For cloning gitolite-admin, you would use:

git clone admin:gitolite-admin

For cloning a repo dilawar has access to:

git clone dilawar:aRepo

See more at "Gitolite: adding user not working, and DENIED by fallthru when cloning as root?".
See also "how gitolite uses ssh"

Adding your public key to the server's ~git/.ssh/authorized_keys file is how ssh uses pubkeys to authenticate users.
Let's say sita@work.station is trying to log in as git@server.
What you have to do is take the ~sita/.ssh/id_rsa.pub file for user sita on work-station and append its contents (remember it's only one line) to ~git/.ssh/authorized_keys for user git on server.

The authorized_keys file can have multiple public keys (from many different people) added to it so any of them can log in to git@server.