why does not ScheduledThreadPoolExecutor provide finite queue?

ScheduledThreadPoolExecutor extends ThreadPoolExecutor, but with less constructor, ScheduledThreadPoolExecutor provide an infinite delayQueue, so that if I submit too many many task, it may cause jvm OOM, why does not it provide a finite queue and reject policy(reject when can't submit new task)?

Why does not it provide a finite queue and reject policy(reject when can't submit new task)?

You would need to ask the designers, but the fact that they decided not to support this suggests that they thought it would not be useful. (Or at least, not useful enough to justify the extra complexity.)

There is another reason. The implementation of ScheduledThreadPoolExecutor depends on the queue returning elements in the correct order. If you look at the source code, you will see that it uses a custom queue class for this. There is a comment that says this:

Using a custom queue (DelayedWorkQueue), a variant of unbounded DelayQueue. The lack of capacity constraint and the fact that corePoolSize and maximumPoolSize are effectively identical simplifies some execution mechanics (see delayedExecute) compared to ThreadPoolExecutor.

So, executor implementation assumes that the delay queue has specific operational characteristic that allow the executor to work properly and efficiently. If you were able to provide your own queue implementation, it probably wouldn't have those characteristics.

Finally, if you want your application to impose a bound on the delay queue size, you could use yourPool.getQueue().length() to check the queue length before scheduling a new task.