apacheamazon-web-servicesserveramazon-elastic-beanstalk

AWS elastic beanstalk 100.0 % of the requests are erroring with HTTP 4xx

My AWS Elastic Beanstalk account keeps getting down with the error: "Environment health has transitioned from Ok to Severe. 100.0 % of the requests are erroring with HTTP 4xx" On a daily basis.

When looking at the server logs, it keeps getting down after access to several odd web pages (which do not exist). Part of the log:

/var/log/httpd/error_log-XXX
[XXX] [:error] [pid XXX] [client XXXX] script '/var/www/html/w.php' not found or unable to stat
[XXX] [:error] [pid XXX] [client XXX] script '/var/www/html/sheep.php' not found or unable to stat
[XXX] [:error] [pid XXX] [client XXX] script '/var/www/html/qaq.php' not found or unable to stat
[XXX] [:error] [pid XXX] [client XXX] script '/var/www/html/db.init.php' not found or unable to stat
[XXX] [:error] [pid XXX] [client XXX] script '/var/www/html/db_session.init.php' not found or unable to stat
[XXX] [:error] [pid XXX] [client XXX] script '/var/www/html/db__.init.php' not found or unable to stat
[XXX] [:error] [pid XXX] [client XXX] script '/var/www/html/wp-admins.php' not found or unable to stat
[XXX] [:error] [pid XXX] [client XXX] script '/var/www/html/m.php' not found or unable to stat
[XXX] [:error] [pid XXX] [XXX] script '/var/www/html/db_dataml.php' not found or unable to stat
...
[XXX] [XXX] [pid XXX] XXX: Graceful restart requested, doing restart

Does anyone know whats going on? Thanks!

Solution

  • I suspect it's some kind of attack (DDoS).

    Maybe someone is scanning your website on different ports and looking for a specific file (w.php) which could be a backdoor or something similar. As the file does not exist it throws errors.

    I recommend the following steps:

    1. Ensure all of your data is backed up on the server.
    2. Re-install the instance from scratch.
    3. Ensure the instance is secured per any CIS benchmark.
    4. Ensure Apache is secured per any Apache CIS benchmark.
    5. Ensure the VPS provider is using an IPS/IDS to monitor your instances, if not find another provider that does.
    6. Ensure that all relevant logs are sent to a central syslog server that is not the same as the web server instance. This will improve the integrity of the logs.
    7. You might want to install the Snort IPS/IDS solution just to see if another attack is launched.
    8. Install a file integrity monitoring solution such as AIDE and monitor config files for changes.

    https://benchmarks.cisecurity.org/downloads/multiform/