Like .mp3, .wav can i able to create my own custom format?. Which can be played only by the custom player i created. But has same quality as .mp3 I don't want my songs copied by others. Those who have my player can play it. Is there any opensource software's available?
You can, but it will be very difficult. You will need to come up with a proprietary audio encoding/compression format, which is still not perfected even by those doing it for decades.
What you should do instead is this. Encrypt an audio file such as an MP3 with your favourite encryption algorithm, and a password only you know. Then in your "custom player", simply decrypt the file before playing it. This way you can encrypt MySong.mp3 to MySong.whatever which is encrypted and unplayable unless you know the password that is embedded into your player.