We are using jdbc sinks (apache flink) with which we are hitting the database maximal session count, especially when we increase parallelism.
Our tests showed that if we increase our default parallelism by setting parallelism.default property more than 1 we have more inactive sessions on the database. When we increase it furthermore, we will come to a point where we can see some exception because jdbc connections cannot established anymore.
By executing our tests with different parallelism.default properties we have encountered following behavior:
application.parallelism.default = 1 active session / inactive session -> 64/35
application.parallelism.default = 2 active session / inactive session -> 66/70
application.parallelism.default = 3 active session / inactive session -> 63/105
application.parallelism.default = 4 active session / inactive session -> 64/140
application.parallelism.default = 5 active session / inactive session -> 64/175
application.parallelism.default = 6 active session / inactive session -> 64/210
application.parallelism.default = 7 active session / inactive session -> 64/245
application.parallelism.default = 8 active session / inactive session -> 64/280
Is there any chance to limit the session used by the jdbc sinks?
We want to know if there is a proper why to keep the inactive sessions under control used by our jdbc sinks.
Instead of changing the default parallelism, just set the parallelism of your JDBC sink to be 1. That way the rest of the workflow can run at a higher parallelism (if needed).