Issue description - Apache logs
I found items similar to this one in the Apache log file:
166.147.68.243 [24/Feb/2013:06:06:25 -0500] 19 web-site.com "-" 408 - "-"
I’ve got custom log format and 408 here stands for status. The log format is:
LogFormat "%h %t %D %V \"%r\" %>s %b \"%{User-agent}i\"" detailed
And normally the line in the log file looks like
184.73.232.108 [26/Feb/2013:08:38:16 -0500] 30677 www.site.com "GET /api/search... HTTP/1.1" 200 205 "Zend_Http_Client"
This is why 408 error lines look strange to me. No request is logged and I have no idea on what should be optimized.
Questions
How to tackle the issue? What additional information or logs should I gather? What might cause the issue? Is this something wrong on the server? Or is this absolutely a network connectivity problem?
I’m addressing this because our customer complained that he has got 408 error on his mobile phone. I found many records in the log file but I have to admit I don’t know what to do with this.
My own research
There are several questions on this subject already here. But people are much more specific. Like they discus issues with some specific client software and scripts. Here I just got the error when opening some page on iPhone.
For example in HTTP, 408 Request timeout, it is suggested to do the GET request before POST. If I have custom client I can do this. But I can not control the behavior of the user’s browser.
Guess #1
When searching the Internet and thinking about the issue I found https://serverfault.com/questions/383290/too-many-408-error-codes-in-access-log
The suggestion is to update the Timeout config parameter back to its default value.
#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 300
I tried the value 30 first because I thought 30 seconds should be enough. But even with 300 seconds default value, I continue to get the errors in the log. I did tail -f when I was writing this text and got more then 10 lines in a few minutes.
To me this does not look a complete solution.
After some studies on the subject I came to the following answer. It is provided by our lead developer and I think it gives good explanation of the subject.
These errors are perfectly normal. They aren't a sign of a larger issue, but normal connections that are holding Apache open for longer than allowed.
For example, client's queries running them over and over kept Apache open. Apache responded by shutting him down appropriately.
If it hadn't, than a handful of people could take over our server and not allow anyone else to connect.
Most often these errors are coming from systems looking for exploits, and you can recreate it by opening a telnet session and leaving it open.
At the same time, tail -f the access log, and within X time (KeepAliveTimeout) you'll see your IP popup with the same error codes.
Back in the days of Apache 1.3, this error was common, but then 2.2 came out and they had it removed until enough of us asked for it to be returned since it give us ideas on how many people are holding open just the port, and not requesting an actual resource, etc.
I think nothing else should be done here except to make sure to set Timeout to some reasonable value as I have described in original question.