wso2 wso2-esb apache-axis wso2-enterprise-integrator mutual-authentication

Mutual SSL in WSO2 EI 6.5.0 - FileNotFoundException - Could not load customSSLProfiles from file path

I am trying Mutual SSL in WSo2 EI by following exact steps mentioned in this blog

WSO2 EI acts as client and Axis2server is backend

Creation of keystore, Exporting certificate, Import the Axis2 certificate into wso2 EI Truststore done successfully by using Java keytool
By using default keystore of WSO2 EI, Exporting certificate and Import the EI certificate into Axis2 Truststore also done.
Configuring hostnames in /etc/hosts file done in Windows 11

Below mentioned changes(https transport receiver) done in <EI_HOME>/samples/axis2Server/repository/conf/axis2.xml

<transportReceiver name="https" class="org.apache.synapse.transport.nhttp.HttpCoreNIOSSLListener">
    <parameter name="port" locked="false">9002</parameter>
    <parameter name="non-blocking" locked="false">true</parameter>
    <parameter name="keystore" locked="false">
        <KeyStore>
            <Location>C:/Development_Avecto/mutualssl/axis2.jks</Location>
            <Type>JKS</Type>
            <Password>axispwd</Password>
            <KeyPassword>axispwd</KeyPassword>
        </KeyStore>
    </parameter>
    <parameter name="truststore" locked="false">
        <TrustStore>
            <Location>C:/Development_Avecto/mutualssl/truststore-axis2.jks</Location>
            <Type>JKS</Type>
            <Password>axispwd</Password>
        </TrustStore>
    </parameter>
    <parameter name="SSLVerifyClient">require</parameter>
</transportReceiver>

Path for Axis2 keystore related files:

Updated: Below mentioned changes (https transport sender) done in <EI_HOME>/conf/axis/axis2.xml

    <transportSender name="https" class="org.apache.synapse.transport.passthru.PassThroughHttpSSLSender">
        <parameter name="non-blocking" locked="false">true</parameter>
        <parameter name="keystore" locked="false">
            <KeyStore>
                <Location>repository/resources/security/wso2carbon.jks</Location>
                <Type>JKS</Type>
                <Password>wso2carbon</Password>
                <KeyPassword>wso2carbon</KeyPassword>
            </KeyStore>
        </parameter>
        <parameter name="truststore" locked="false">
            <TrustStore>
                <Location>repository/resources/security/client-truststore.jks</Location>
                <Type>JKS</Type>
                <Password>wso2carbon</Password>
            </TrustStore>
        </parameter>
        <parameter name="HostnameVerifier">AllowAll</parameter>
        <parameter name="dynamicSSLProfilesConfig">
 <filePath>repository/deployment/server/mutual_ssl_profiles.xml</filePath>
 <fileReadInterval>3600000</fileReadInterval>  
</parameter>
</transportSender>

Path for EI keystore related files:

multi_ssl_profiles.xml:

    <parameter name="customSSLProfiles">
<profile>
 <servers>axis2.backend.mytest:9002</servers>
 <TrustStore>
 <Location>C:/Development_Avecto/mutualssl/wso2ei-6.5.0/repository/resources/security/client-truststore.jks</Location>
 <Type>JKS</Type>
 <Password>wso2carbon</Password>
 </TrustStore>
 <KeyStore>
 <Location>C:/Development_Avecto/mutualssl/wso2ei-6.5.0/repository/resources/security/wso2carbon.jks</Location>
 <Type>JKS</Type>
 <Password>wso2carbon</Password>
 <KeyPassword>wso2carbon</KeyPassword>
 </KeyStore>
</profile>
</parameter>

Path for multi_ssl_profile.xml :

Axis2server.bat started and running
When starts EI Server got below ERROR in logs.

[2023-04-02 19:36:08,661] [] ERROR - ClientConnFactoryBuilder FileNotFoundException - Could not load customSSLProfiles from file path: C:/Development_Avecto/mutualssl/wso2ei-6.5.0/repository/deployment/server/mutual_ssl_profiles.xml java.io.FileNotFoundException: C:\Development_Avecto\MUTUAL~1\WSO2EI~2.0\C:\Development_Avecto\mutualssl\wso2ei-6.5.0\repository\deployment\server\mutual_ssl_profiles.xml (The filename, directory name, or volume label syntax is incorrect)
I cross verified all file paths in my system which are correctly configured in server config files.

Updated-WSO2Carbon.log:

    C:\Development_Avecto\mutualssl\wso2ei-6.5.0\bin>integrator.bat
JAVA_HOME environment variable is set to C:\Program Files\Java\jdk1.8.0_291
CARBON_HOME environment variable is set to C:\Development_Avecto\MUTUAL~1\WSO2EI~2.0\bin\..
[2023-04-03 15:43:14,000] []  INFO - CarbonCoreActivator Starting WSO2 Carbon...
[2023-04-03 15:43:14,009] []  INFO - CarbonCoreActivator Operating System : Windows 10 10.0, amd64
[2023-04-03 15:43:14,009] []  INFO - CarbonCoreActivator Java Home        : C:\Program Files\Java\jdk1.8.0_291\jre
[2023-04-03 15:43:14,010] []  INFO - CarbonCoreActivator Java Version     : 1.8.0_291
[2023-04-03 15:43:14,010] []  INFO - CarbonCoreActivator Java VM          : Java HotSpot(TM) 64-Bit Server VM 25.291-b10,Oracle Corporation
[2023-04-03 15:43:14,011] []  INFO - CarbonCoreActivator Carbon Home      : C:\Development_Avecto\MUTUAL~1\WSO2EI~2.0\bin\..
[2023-04-03 15:43:14,011] []  INFO - CarbonCoreActivator Java Temp Dir    : C:\Development_Avecto\MUTUAL~1\WSO2EI~2.0\bin\..\wso2\tmp
[2023-04-03 15:43:14,011] []  INFO - CarbonCoreActivator User             : 111002, en-IN, Asia/Calcutta
[2023-04-03 15:43:14,228] []  INFO - DefaultCryptoProviderComponent 'CryptoService.Secret' property has not been set. 'org.wso2.carbon.crypto.provider.SymmetricKeyInternalCryptoProvider' won't be registered as an internal crypto provider. Please set the secret if the provider needs to be registered.
[2023-04-03 15:43:14,255] []  INFO - GoogleTokenGenDSComponent Activating GoogleTokengen DS component
[2023-04-03 15:43:14,393] []  INFO - KafkaEventAdapterServiceDS Successfully deployed the Kafka output event adaptor service
[2023-04-03 15:43:17,625] []  INFO - EmbeddedRegistryService Configured Registry in 65ms
[2023-04-03 15:43:17,709] []  INFO - RegistryCoreServiceComponent Registry Mode    : READ-WRITE
[2023-04-03 15:43:21,390] []  INFO - SolrClient Default Embedded Solr Server Initialized
[2023-04-03 15:43:21,681] []  INFO - UserStoreMgtDSComponent Carbon UserStoreMgtDSComponent activated successfully.
[2023-04-03 15:43:38,787] []  INFO - TaglibUriRule TLD skipped. URI: http://tiles.apache.org/tags-tiles is already defined
[2023-04-03 15:43:40,043] []  INFO - ClusterBuilder Clustering has been disabled
[2023-04-03 15:43:40,415] []  INFO - UserStoreConfigurationDeployer User Store Configuration Deployer initiated.
[2023-04-03 15:43:40,416] []  INFO - UserStoreConfigurationDeployer User Store Configuration Deployer initiated.
[2023-04-03 15:43:41,828] []  INFO - VFSTransportSender VFS Sender started
[2023-04-03 15:43:41,887] []  INFO - PassThroughHttpSender Initializing Pass-through HTTP/S Sender...
[2023-04-03 15:43:41,948] []  INFO - PassThroughHttpSender Pass-through HTTP Sender started...
[2023-04-03 15:43:41,949] []  INFO - PassThroughHttpSSLSender Initializing Pass-through HTTP/S Sender...
[2023-04-03 15:43:41,964] [] ERROR - ClientConnFactoryBuilder FileNotFoundException - Could not load customSSLProfiles from file path: repository/deployment/server/mutual_ssl_profiles.xml
java.io.FileNotFoundException: C:\Development_Avecto\MUTUAL~1\WSO2EI~2.0\repository\deployment\server\mutual_ssl_profiles.xml (The system cannot find the file specified)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at java.io.FileInputStream.<init>(FileInputStream.java:93)
        at org.apache.axiom.om.impl.builder.StAXOMBuilder.<init>(StAXOMBuilder.java:148)
        at org.apache.synapse.transport.nhttp.config.ClientConnFactoryBuilder.loadDynamicSSLConfig(ClientConnFactoryBuilder.java:527)
        at org.apache.synapse.transport.nhttp.config.ClientConnFactoryBuilder.getCustomSSLContexts(ClientConnFactoryBuilder.java:229)
        at org.apache.synapse.transport.nhttp.config.ClientConnFactoryBuilder.parseSSL(ClientConnFactoryBuilder.java:195)
        at org.apache.synapse.transport.passthru.PassThroughHttpSSLSender.initConnFactoryBuilder(PassThroughHttpSSLSender.java:45)
        at org.apache.synapse.transport.passthru.PassThroughHttpSender.init(PassThroughHttpSender.java:167)
        at org.apache.synapse.transport.passthru.PassThroughHttpSSLSender.init(PassThroughHttpSSLSender.java:33)
        at org.apache.axis2.context.ConfigurationContextFactory.initTransportSenders(ConfigurationContextFactory.java:300)
        at org.apache.axis2.context.ConfigurationContextFactory.init(ConfigurationContextFactory.java:231)
        at org.apache.axis2.context.ConfigurationContextFactory.createConfigurationContext(ConfigurationContextFactory.java:93)
        at org.wso2.carbon.core.CarbonConfigurationContextFactory.createNewConfigurationContext(CarbonConfigurationContextFactory.java:65)
        at org.wso2.carbon.core.init.CarbonServerManager.initializeCarbon(CarbonServerManager.java:401)
        at org.wso2.carbon.core.init.CarbonServerManager.start(CarbonServerManager.java:220)
        at org.wso2.carbon.core.internal.CarbonCoreServiceComponent.activate(CarbonCoreServiceComponent.java:105)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
        at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
        at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345)
        at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
        at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
        at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
        at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
        at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
        at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
        at org.eclipse.equinox.http.servlet.internal.Activator.registerHttpService(Activator.java:81)
        at org.eclipse.equinox.http.servlet.internal.Activator.addProxyServlet(Activator.java:60)
        at org.eclipse.equinox.http.servlet.internal.ProxyServlet.init(ProxyServlet.java:40)
        at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.init(DelegationServlet.java:38)
        at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1230)
        at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1174)
        at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1066)
        at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5433)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5731)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1707)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1697)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[2023-04-03 15:43:41,971] [] FATAL - CarbonServerManager WSO2 Carbon initialization Failed
java.lang.NullPointerException
        at org.apache.synapse.transport.nhttp.config.ClientConnFactoryBuilder.getCustomSSLContexts(ClientConnFactoryBuilder.java:231)
        at org.apache.synapse.transport.nhttp.config.ClientConnFactoryBuilder.parseSSL(ClientConnFactoryBuilder.java:195)
        at org.apache.synapse.transport.passthru.PassThroughHttpSSLSender.initConnFactoryBuilder(PassThroughHttpSSLSender.java:45)
        at org.apache.synapse.transport.passthru.PassThroughHttpSender.init(PassThroughHttpSender.java:167)
        at org.apache.synapse.transport.passthru.PassThroughHttpSSLSender.init(PassThroughHttpSSLSender.java:33)
        at org.apache.axis2.context.ConfigurationContextFactory.initTransportSenders(ConfigurationContextFactory.java:300)
        at org.apache.axis2.context.ConfigurationContextFactory.init(ConfigurationContextFactory.java:231)
        at org.apache.axis2.context.ConfigurationContextFactory.createConfigurationContext(ConfigurationContextFactory.java:93)
        at org.wso2.carbon.core.CarbonConfigurationContextFactory.createNewConfigurationContext(CarbonConfigurationContextFactory.java:65)
        at org.wso2.carbon.core.init.CarbonServerManager.initializeCarbon(CarbonServerManager.java:401)
        at org.wso2.carbon.core.init.CarbonServerManager.start(CarbonServerManager.java:220)
        at org.wso2.carbon.core.internal.CarbonCoreServiceComponent.activate(CarbonCoreServiceComponent.java:105)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
        at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
        at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345)
        at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
        at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
        at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
        at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
        at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
        at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214)
        at org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433)
        at org.eclipse.equinox.http.servlet.internal.Activator.registerHttpService(Activator.java:81)
        at org.eclipse.equinox.http.servlet.internal.Activator.addProxyServlet(Activator.java:60)
        at org.eclipse.equinox.http.servlet.internal.ProxyServlet.init(ProxyServlet.java:40)
        at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.init(DelegationServlet.java:38)
        at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1230)
        at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1174)
        at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1066)
        at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5433)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5731)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1707)

UPDATED- CARBON_HOME:

Why customSSLProfiles configured in mutual_ssl_profiles.xml is not loaded or Did I make any mistake while specifying file path?

Please clarify on the same.

Solution

First of all, the error The process cannot access the file because it is being used by another process. indicates that the files are being used by a different process. So I assume the Axis2 Server you started is locking some dependencies that are common to both EI and Axis2Server. So not sure it would affect the runtim. In order to get around this, create a copy of WSO2 EI (C:\Development_Avecto\mutualssl\wso2ei-6.5.0) and start the Axis2 server from there.

Regarding the SSlProfile error, it seems the Path is incorrect. C:\Development_Avecto\MUTUAL~1\WSO2EI~2.0\C:\Development_Avecto\mutualssl\wso2ei-6.5.0\repository\deployment\server\mutual_ssl_profiles.xml As you can see CARBON_HOME(C:\Development_Avecto\MUTUAL~1\WSO2EI~2.0) seems to be appended to your path, so this means the server didn't recognize it as an absolute path. Hence try giving a relative path in the axis2 configs like repository\deployment\server\mutual_ssl_profiles.xml.

Als o I noticed the CARBON_HOME (C:\Development_Avecto\MUTUAL~1\WSO2EI~2.0\) is not the same as the actual server location (C:\Development_Avecto\mutualssl\wso2ei-6.5.0). You may have to fix that as well.

Update

After all it doesn;t really seems like a issue with the CARBON_HOME, that seems to be the way CARBON_HOME is logged on windows. (No idea why). These are the configs that worked for me.

In axis2.xml

<parameter name="dynamicSSLProfilesConfig">
         <filePath>repository/deployment/server/mutual_ssl_profiles.xml</filePath>
         <fileReadInterval>3600000</fileReadInterval>  
</parameter>

mutual_ssl_profiles.xml file content

<parameter name="customSSLProfiles">
    <profile>
         <servers>axis2.backend.mytest:9002</servers>
         <TrustStore>
         <Location>repository/resources/security/client-truststore.jks</Location>
         <Type>JKS</Type>
         <Password>wso2carbon</Password>
         </TrustStore>
         <KeyStore>
         <Location>repository/resources/security/wso2carbon.jks</Location>
         <Type>JKS</Type>
         <Password>wso2carbon</Password>
         <KeyPassword>wso2carbon</KeyPassword>
         </KeyStore>
    </profile>
</parameter>

Once started the wso2carbon.log

 INFO - ClientConnFactoryBuilder customSSLProfiles configuration is loaded from path: C:\soft\wso2ei\WSO2EI~1.0_S\repository/deployment/server/mutual_ssl_profiles.xml