SSL slow. Establishing secure connection taking too long

I have a dedicated server with 256GB RAM 6 CPUs (12 Threads) on Hetzner, and it is located in Germany. I have CENTOS 7.5. EA4.

My problem is with SSL. Every day for about 2 hours, we have 40 requests in one second and finishing requests takes about 20 seconds. Non-SSL takes 0.5 or less. Here is an example.

From 13:00 to 15:30 (UTC+4), SSL requests take the most time. The problem is evident when you open this link with SSL and without.

I have WHM available. I've noticed ModSecurity and wonder if it might be the problem. I've applied most of the settings provided here, but there is not much regarding SSL.

In case certificates are the reason for all of this:

Thank you for your answers guys.

It was not OCSP after all. There happened to be some issues with certificates and some Apache configuration. We hired server guy and he fixed it.

So if anyone has this kind of problem should check Server configuration and look for ways to optimize it, also check for certificates. This fixed the 3-4 second of the waiting time on every response.

Bigger issue was use of geoplugin for detecting Country/City from IP address. I didn't know Curl could slow down response times that low. I'm not blaming geoplugin of course. When I profiled my Code it said 127 Miliseconds from start till the end but turns out that profiler just skipped this geoplugin waiting time or smth.

In conclusion, modifying Code, dealing with certificates and server configuration made it happen.

P.S. I don't know how to deal with this bounty. I don't want it to go waste so I'm gonna give it to someone who answered even tho answer didn't resolve my issue and question was answered day before bounty was expiring and issue was already resolved.