I have a task to implement pretty rare symmetric block cipher. It's unavailable in popular crypto providers like Bouncy Castle, Sun's JCE and even in GNU Crypto, so I can't use JCE approach.
I got Java sources of this cipher for encryption/decryption of single block, but can't find where to find CTR or GCM - Galois/Counter Mode implementations. Can anyone help me to find them?
It's probably best to first get the block cipher to adhere to the BlockCipher interface of the lightweight Bouncy Castle library. When you've extended that then you can simply insert it into the constructor of GCMBlockCipher. Beware that the block cipher must have a block size of 128 bit - 16 bytes - for it to work in GCM mode.
You can do the same thing for CTR mode, and it may be better to start with that as the mode is much simpler so you won't get to do as much debugging if it fails for some reason. You won't find CTRBlockCipher as mode of operation though. The reason for that is that CTR is/was known under different names. You'll have to use the SICBlockCipher instead.
As the documentation states:
Implements the Segmented Integer Counter (SIC) mode on top of a simple block cipher. This mode is also known as CTR mode.
You can use your own package name; as you can see the required functionality is "public" so extensions are possible.
Once you've got that going you can try and register the cipher and mode into the Bouncy Castle provider, or you can create your own provider. Creating a full CipherSpi (service provider implementation) is mainly useful if you want to use the block cipher in code that requires an instance of Java's Cipher class.