java opensearch apache-httpclient-5.x java-security-manager javapolicy

Apache HTTP Client 5 throws java.security.AccessControlException, policy file is specified

I'm writing an OpenSearch plugin in Java, using the provided template.

Except communication with OpenSearch Dashboards which works fine, I need to implement communication with external service. Currently, it runs locally at localhost:18880 and is available (e.g. via Postman). I decided to use Apache Http Client for this purpose:

dependencies {
    implementation 'org.apache.httpcomponents.core5:httpcore5:5.3.1'
    implementation 'org.apache.httpcomponents.core5:httpcore5-h2:5.3.1'
    implementation 'org.apache.httpcomponents.client5:httpclient5:5.3.1'
}

However when I try to call it inside my plugin:

public static ServiceResponse<Boolean> busy() throws Exception {
    URI uri = new URIBuilder("localhost:18880")
            .appendPath("busy")
            .build();

    try (CloseableHttpClient httpClient = HttpClients.createDefault()) {
        HttpGet request = new HttpGet(uri);

        try (ClassicHttpResponse response = httpClient.executeOpen(null, request, null)) {
            // ...

I get java.security.AccessControlException: access denied ("java.net.SocketPermission" "127.0.0.1:18880" "connect,resolve"). Full stack trace is:

java.security.AccessControlException: access denied ("java.net.SocketPermission" "127.0.0.1:18880" "connect,resolve")
    at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488) ~[?:?]
    at java.base/java.security.AccessController.checkPermission(AccessController.java:1071) ~[?:?]
    at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
    at java.base/java.lang.SecurityManager.checkConnect(SecurityManager.java:905) ~[?:?]
    at java.base/java.net.Socket.connect(Socket.java:747) ~[?:?]
    at org.apache.hc.client5.http.socket.PlainConnectionSocketFactory.lambda$connectSocket$0(PlainConnectionSocketFactory.java:91) ~[httpclient5-5.3.1.jar:5.3.1]
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
    at org.apache.hc.client5.http.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:90) ~[httpclient5-5.3.1.jar:5.3.1]
    at org.apache.hc.client5.http.socket.ConnectionSocketFactory.connectSocket(ConnectionSocketFactory.java:123) ~[httpclient5-5.3.1.jar:5.3.1]
    at org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:189) ~[httpclient5-5.3.1.jar:5.3.1]
    at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:450) ~[httpclient5-5.3.1.jar:5.3.1]
    at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:162) ~[httpclient5-5.3.1.jar:5.3.1]
    at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:172) ~[httpclient5-5.3.1.jar:5.3.1]
    at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:142) ~[httpclient5-5.3.1.jar:5.3.1]
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.3.1.jar:5.3.1]
    at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192) ~[httpclient5-5.3.1.jar:5.3.1]
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.3.1.jar:5.3.1]
    at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:113) ~[httpclient5-5.3.1.jar:5.3.1]
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.3.1.jar:5.3.1]
    at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:152) ~[httpclient5-5.3.1.jar:5.3.1]
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.3.1.jar:5.3.1]
    at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:116) ~[httpclient5-5.3.1.jar:5.3.1]
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.3.1.jar:5.3.1]
    at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170) ~[httpclient5-5.3.1.jar:5.3.1]
    at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:87) ~[httpclient5-5.3.1.jar:5.3.1]
    at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:55) ~[httpclient5-5.3.1.jar:5.3.1]
    at org.apache.hc.client5.http.classic.HttpClient.executeOpen(HttpClient.java:183) ~[httpclient5-5.3.1.jar:5.3.1]
    at at.risedev.mlbase.idp.ad.service.external.AnomalyDetectionService.busy(AnomalyDetectionService.java:423) ~[IdpAdOpenSearchPlugin-2.19.1.0-SNAPSHOT.jar:2.19.1.0-SNAPSHOT]
    at at.risedev.mlbase.idp.ad.service.PollService.pollAdService(PollService.java:55) [IdpAdOpenSearchPlugin-2.19.1.0-SNAPSHOT.jar:2.19.1.0-SNAPSHOT]
    at at.risedev.mlbase.idp.ad.rest.RestPollAdServiceAction.lambda$prepareRequest$0(RestPollAdServiceAction.java:69) [IdpAdOpenSearchPlugin-2.19.1.0-SNAPSHOT.jar:2.19.1.0-SNAPSHOT]
    at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:127) [opensearch-2.19.1.jar:2.19.1]
    at org.opensearch.rest.RestController.dispatchRequest(RestController.java:381) [opensearch-2.19.1.jar:2.19.1]
    at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:467) [opensearch-2.19.1.jar:2.19.1]
    at org.opensearch.rest.RestController.dispatchRequest(RestController.java:287) [opensearch-2.19.1.jar:2.19.1]
    at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:374) [opensearch-2.19.1.jar:2.19.1]
    at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:482) [opensearch-2.19.1.jar:2.19.1]
    at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:357) [opensearch-2.19.1.jar:2.19.1]
    at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:56) [transport-netty4-client-2.19.1.jar:2.19.1]
    at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42) [transport-netty4-client-2.19.1.jar:2.19.1]
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:72) [transport-netty4-client-2.19.1.jar:2.19.1]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:120) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289) [netty-handler-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:697) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:660) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) [netty-common-4.1.118.Final.jar:4.1.118.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.118.Final.jar:4.1.118.Final]
    at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

Which indicates exception is thrown from httpClient.executeOpen() method of code above.

To run, I use ./gradlew run which is defined in build.gradle:

testClusters.integTest {
    testDistribution = "INTEG_TEST"
    
    // This installs our plugin into the testClusters
    plugin(project.tasks.bundlePlugin.archiveFile)
}

run {
    useCluster testClusters.integTest
}

1. I tried to disable SecurityManager in app

testClusters.integTest {
    testDistribution = "INTEG_TEST"
    systemProperty 'java.security.manager', 'false'

    // This installs our plugin into the testClusters
    plugin(project.tasks.bundlePlugin.archiveFile)
}

But application fails to start:

> Task :run FAILED
Exec output and error:
| Output for ./bin/opensearch-plugin:warning: no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release
| Error occurred during initialization of VM
| java.lang.InternalError: Could not create SecurityManager
|       at java.lang.System.initPhase3(java.base@21.0.6/System.java:2305)
| Caused by: java.lang.ClassNotFoundException: false
|       at jdk.internal.loader.BuiltinClassLoader.loadClass(java.base@21.0.6/BuiltinClassLoader.java:641)
|       at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(java.base@21.0.6/ClassLoaders.java:188)
|       at java.lang.ClassLoader.loadClass(java.base@21.0.6/ClassLoader.java:526)
|       at java.lang.Class.forName0(java.base@21.0.6/Native Method)
|       at java.lang.Class.forName(java.base@21.0.6/Class.java:534)
|       at java.lang.Class.forName(java.base@21.0.6/Class.java:513)
|       at java.lang.System.initPhase3(java.base@21.0.6/System.java:2289)

I also tried other options (disallow, null, default, <empty string>) for java.security.manager with same result

2. I tried to extend security policy

testClusters.integTest {
    testDistribution = "INTEG_TEST"
    systemProperty 'java.security.policy', rootProject.file("src/main/resources/main.policy").absolutePath

    // This installs our plugin into the testClusters
    plugin(project.tasks.bundlePlugin.archiveFile)
}

Full contents of main.policy file is:

grant {
    permission java.net.SocketPermission "localhost:*", "connect,resolve";
}

I can verify in logs JVM options are passed correctly from logs when running ./gradlew run:

[2025-05-20T14:20:13.115313433Z] [BUILD] Starting OpenSearch process
[2025-05-20T16:20:16,321][INFO ][o.o.n.Node               ] [integTest-0] version[2.19.1], pid[36612], build[zip/2e4741fb45d1b150aaeeadf66d41445b23ff5982/2025-02-27T01:16:47.726162386Z], OS[Linux/6.13.8-200.fc41.x86_64/amd64], JVM[Red Hat, Inc./OpenJDK 64-Bit Server VM/21.0.6/21.0.6+7]
[2025-05-20T16:20:16,324][INFO ][o.o.n.Node               ] [integTest-0] JVM home [/usr/lib/jvm/java-21-openjdk]
[2025-05-20T16:20:16,330][INFO ][o.o.n.Node               ] [integTest-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/home/kaorise/Documents/work/idp-ad-opensearch-plugin/backend/build/testclusters/integTest-0/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=logs, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, --add-modules=jdk.incubator.vector, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Xms512m, -Xmx512m, -ea, -esa, -Djava.security.policy=/home/kaorise/Documents/work/idp-ad-opensearch-plugin/backend/src/main/resources/main.policy, -Djava.security.manager=allow, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/home/kaorise/Documents/work/idp-ad-opensearch-plugin/backend/build/testclusters/integTest-0/distro/2.19.1-INTEG_TEST, -Dopensearch.path.conf=/home/kaorise/Documents/work/idp-ad-opensearch-plugin/backend/build/testclusters/integTest-0/config, -Dopensearch.distribution.type=zip, -Dopensearch.bundled_jdk=false]

JVM args contain both of my specified arguments. However issue persists

What is proper way to configure policies for opensearch plugin. It seems like my policy file is ignored. Or is issue in something else?

Solution

Migrating my plugin to OpenSearch version 3.0.0 did not resolve issue automatically, but helped me to debug it due to improved logging.

New logs include lines:

...
[2025-05-21T17:02:12,142][WARN ][stderr] [integTest-0] java.security.policy: error parsing file:/home/.../plugin/backend/src/main/resources/main.policy:
[2025-05-21T17:02:12,145][WARN ][stderr] [integTest-0]  expected [;], read [end of file]
...

Which confirmed the fact that main.policy file located at src/main/resources/ is discovered but fails to parse. The logs indicated that the problem was in missing semicolon in policy file. Correct version is:

grant {
    permission java.net.SocketPermission "localhost:*", "connect,resolve";
};

with semicolon after parentheses.

To recap

As mentioned by @user207421, specifying .policy in gradlew build wouldn't work because policy from src is not included in build and won't work with plugin loaded as jar into existing OpenSearch instance. By inspecting other plugins like Anomaly Detection plugin, it is seen that plugin-security.policy should be created at src/main/plugin-metadata. Specifying policy that way works without any other modifications to gradlew.build or JVM environment. I verified that plugin-metadata is included to runtime, and tested on non-local environment.

Ensure syntax of .policy file is correct

grant {
    permission java.net.SocketPermission "localhost:*", "connect,resolve";
};

Thanks to all who commented, your comments helped me to debug and locate the issue