Need help on an intermittent headache. The code is calling com.google.api.client.http.HttpRequest#executeAsync() which basically has the following logic,
@Beta
public Future<HttpResponse> executeAsync(Executor executor) {
FutureTask<HttpResponse> future = new FutureTask<HttpResponse>(new Callable<HttpResponse>() {
public HttpResponse call() throws Exception {
return execute();
}
});
executor.execute(future);
return future;
}
@Beta
public Future<HttpResponse> executeAsync() {
return executeAsync(Executors.newSingleThreadExecutor());
}
The call runs into java.util.concurrent.RejectedExecutionException sometimes and from the log seems when this happens it's always accompanied by garbage collection. Below is a sample of the log pattern when this happens,
2017-09-26 11:04:56.039186 2017-09-26T11:04:56.012+0000: [GC pause (G1 Evacuation Pause) (young) 213M->50M(300M), 0.0262262 secs]
2017-09-26 11:04:56.048210 java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@71a0a39 rejected from java.util.concurrent.ThreadPoolExecutor@36c306aa[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2017-09-26 11:04:56.048212 at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_141]
2017-09-26 11:04:56.048214 at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_141]
2017-09-26 11:04:56.048216 at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_141]
2017-09-26 11:04:56.048218 at java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668) ~[?:1.8.0_141]
2017-09-26 11:04:56.048220 at com.google.api.client.http.HttpRequest.executeAsync(HttpRequest.java:1085) ~[google-http-client-1.21.0.jar:1.21.0]
2017-09-26 11:04:56.048222 at com.google.api.client.http.HttpRequest.executeAsync(HttpRequest.java:1099) ~[google-http-client-1.21.0.jar:1.21.0]
Based on my understanding GC shouldn't clean up this executor since it hasn't gone out of scope yet, but judging by the log of the error this seems to be the behavior.
EDIT: Thanks for the quick responses. I probably need to clarify I cannot reproduce this either at my will otherwise I probably will have more clue. The code generally runs fine but this error does happen rarely and all we have is the log information I pasted for the symptom.
EDIT2: Clarification the code I pasted is not mine. It's the open source library google-http-java-client I'm using. So I have no way to change that. I sure can instead create a long-term Executor myself and call it with the first method, but I'm trying to understand what's the problem for now.
The ExecutorService returned by Executors#newSingleThreadExecutor() happens to be ThreadPoolExecutor wrapped in a FinalizableDelegatedExecutorService, an implementation detail that shuts down its wrapped ExecutorService on finalization. We can tell the executor is shutdown from your logs
[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
Why is the object being finalized, though? You state
Based on my understanding GC shouldn't clean up this executor since it hasn't gone out of scope yet, but judging by the log of the error this seems to be the behavior.
This is a common misconception. Scope is a compile time feature that determines where a name can be used to refer to some entity in source code. It has no bearing on runtime garbage collection.
Garbage collection (and finalization) is controlled by reachability of objects. The Java Language Specification states
When an object is no longer referenced, it may be reclaimed by the garbage collector. If an object declares a finalizer, the finalizer is executed before the object is reclaimed to give the object a last chance to clean up resources that would not otherwise be released. When a class is no longer needed, it may be unloaded.
The JVM will not garbage collect reachable objects
A reachable object is any object that can be accessed in any potential continuing computation from any live thread.
As explained in the accepted answer (by an Oracle developer) for this question
reachability analysis allows
for an object to be finalized and garbage collected even if there are references to it in local variables on the stack
What you're seeing is the JVM making the decision that the FinalizableDelegatedExecutorService (and its ThreadPoolExecutor) is no longer reachable by a continuing computation from a live thread and finalizing it. That action shuts down the executor and the RejectedExecutionException is thrown when a task is submitted.
This is a known issue for that implementation and bug JDK-8145304 was opened to discuss solutions (essentially just better documentation).
If it was up to me, google-http-client would change their implementation to use Executors.newFixedThreadPool(1) which isn't wrapped with a FinalizableDelegatedExecutorService. (I've opened this issue to discuss a solution for the library.)
My suggestion is that you create your own ExecutorService (maybe with newFixedThreadPool) and use the overloaded executeAsync(Executor) method.
The issue has been fixed, see here.