While testing my FIPS 140-2 compliance configuration, it appears that I am still able to access the unapproved MD5 algorithm. This leads me to believe that I have configured the BouncyCastle FIPS JCE provider incorrectly. The following code does not behave as I expect:
System.setProperty("org.bouncycastle.fips.approved_only", "true");
Security.insertProviderAt(new BouncyCastleFipsProvider(), 1);
System.out.println(CryptoServicesRegistrar.isInApprovedOnlyMode());
MessageDigest digest = MessageDigest.getInstance("MD5", "BCFIPS");
System.out.println(digest.getAlgorithm());
System.out.println(digest.getProvider());
Cipher.getInstance("DES", "BCFIPS");
And the output of that being:
true
MD5
BCFIPS version 1.000203
This code throws a NoSuchAlgorithmException when it tries to get the DES cipher just as I expect since it is not a FIPS 140-2 approved algorithm. If I remove the 'approved_only' property, the DES cipher instance is created successfully. However I would think that in approved only mode the MD5 MessageDigest should also fail. While it is a bit old (2015), page 6 of https://www.bouncycastle.org/fips/BCFipsDescription.pdf states that MD5 should not be available in approved mode, and more recently page 5 of https://csrc.nist.gov/csrc/media/publications/fips/140/2/final/documents/fips1402annexa.pdf does not list MD5 in its approved list of secure hash standards.
Additionally, other hashing algorithms from https://www.bouncycastle.org/fips/BCFipsDescription.pdf do not work in approved mode but are available in unapproved mode such as WHIRLPOOL. I haven't tested all of them but so far MD5 is the only unapproved algorithm that I can get to work in approved mode.
My question is: why am I able to create a MD5 MessageDigest in approved mode? Is there something wrong with how I'm implementing the BouncyCastle JCE?
https://downloads.bouncycastle.org/fips-java/BC-FJA-SecurityPolicy-1.0.2.pdf lists MD5 under "Table 7 - Non-Approved but Allowed Cryptographic Functions" for use within TLS only.
But apparently the MD5 implementation is then made available generally since it doesn't know whether it's being used within the context of TLS.
In https://github.com/bcgit/bc-java/issues/1282#issuecomment-1325782487, David Hook (one of the Bouncy Castle founders) says:
That's correct the MD5 implementation doesn't have any context, but it's available for TLS.
As far the Spring Boot app goes, you can't block MD5 automatically, you'll need to rely on telling your developers not to do it. If you want FIPS compliance the module can only be used in accordance with the security policy.
I guess the other thing I should add, which might help (or might not) it's not the use of MD5 outside of TLS that is the problem, it's the use of MD5 outside of TLS as a cryptographic message digest that's the problem. For example, say someone was using MD5 in a Bloom filter, that would be allowed, in that case it's just a hash function.