I am trying to encrypt a Saml assertion with the public certificate provided by the vendor can someone provide the correct way of encryption. What encryption method is used for encrypting XML. Any suggestion will be really helpful.
Encryption of SAML Assertions is specified in the SAML specification as mentioned by Bernhard.
The Assertion is then instead represented by an EncryptedAssertion element. To encrypt the assertion you will need some software. From the sounds of your post it sound like you are doing it manually. In this case the OpenSAML library can be of use.
My blog offers a lot of information on how to use OpenSAML. I do not have a post on encryption. Encryption with OpenSAML is shown in my book though, A Guide to OpenSAML .