Server load is minimal but website responds poorly

I have VPS on hetzner. Server is located in Germany.

It has 256GB RAM, 6CPUs (12 threads).

I have a file which since yesterday, is requested about 30 times in one second. File has 2 Select, 2 Update, 2 Insert queries, so I assumed (not sure how this works) from this file server has about 180 requests per second. So right after this requests started, all the websites on the server just started loading poorly. I made this file run just one select query and than die. This didn't help. In WHM load is aboiut 0.02.

I've checked for error logs and there is no max_user_connection or any error there.

I have enabled slow query log and checked log file. there is nothing (I've tested it with select sleep(10) and this query was logged).

This is visit statistics, please bring your attention to may 30th:

Bandwidth stats for last 24 hours:

There are many errors like this in ssl_log (diff IPs of course):

188.121.206.150 - - [30/May/2018:19:50:03 +0200] "-" 408 - "-" "-"

I've been searching web a lot and couldn't find any solution. Could anyone at least tell what should I monitor or where. I have full access to anything there is possible inside the server. Any help is appreciated.

UPDATE 1

I have subdomain: banners.analyticson.com (access allowed for now) and there I have all the images and html5 files that are requested. Take one image for example : https://banners.analyticson.com/img/suy8G1S6RU.jpg

It needs too much time to load. As I noticed, this sub domain has some issue. Script, that I mentioned earlier (with 6 queries) just tries to get one of those banners to the user, so result of that script is to return one banner from banners.analyticson.com.

UPDATE 2

I've checked my script, it is fine. It takes less than 1 second to complete. I've also checked Top command and there is a result. I'm not sure if $MEM value is fine.

You're going to have to narrow the problem down...

There are multiple potential issues.

First thing to eliminate would be the performance of your new script on a development laptop - I assume you're using PHP, so use the profiling tools to work out what is going on. If it's a database query, you'll see which one by looking at the profiler.

If your PHP script and database queries are fine, the next thing to look at: it sounds like you've hit some bottleneck resource on your infrastructure. In these cases, scripts that run fine as a single request start queueing for the bottleneck resource, and every new request adds to the queue until the whole server starts to crawl. This can be a bit of a puzzle - start with top and keep digging.

Next, I'd look at configuration of Apache to make sure everything is squeaky clean - Apache used to have a default to do a reverse DNS lookup for every request, which slows the server down rather impressively on production. You may also want to look at your SSL configuration - the error you report is linked to a load balancer issue.

If it's not as simple as memory, CPU etc., you're into more esoteric issues. You may need to ramp up a load testing rig so you can experiment without affecting the live site - typically, I do this on a machine as similar to live as possible, using Apache JMeter to generate load, and find the "inflection point". Typically, you see response times increase linearly with the number of concurrent requests, until you hit the bottleneck resource, at which point the response time increases rapidly. As a simple example, if you have 10 database connections available, response time should increase linearly up to 10 concurrent connections, and then become much larger from 11 up.

Knowing where the inflection point is and being able to recreate it allows you to use PHP profiling tools under load. This is a lot of work.

UPDATE

You're using php-cgi; this is easily the most inefficient way of running PHP scripts. Your server is barely breaking a sweat - CPU and memory basically idle. Here's a comparison for how to run PHP; consider changing to mod_php.