I am facing a problem where my AWS Elastic Beanstalk environment's health has been degraded automatically. I received the following error messages in the Environment Events log:Failed to deploy configuration.
August 15, 2024 13:07:33 (UTC+5)
ERROR
Rollback of environment Univexch-env failed. Reason: The following resource(s) failed to update: [AWSEBRDSDatabase].
ERROR
Updating RDS database named: awseb-e-i4w84yjffc-stack-awsebrdsdatabase failed.
Reason: Resource handler returned message:
"User: arn:aws:sts::14:assumed-role/aws-elasticbeanstalk-service-role/elasticbeanstalk
is not authorized to perform: rds:ModifyDBInstance on resource:
arn:aws:rds:us-east-1:141408100488:db:awseb-e-i4w84yjffc-stack-awsebrdsdatabase
because no identity-based policy allows the rds:ModifyDBInstance action
(Service: Rds, Status Code: 403, Request ID: 43f16117-cf09-4a3c-b4c7-9f4dfe5e)"
(RequestToken: 336c4947-d7da-c10b-3360-dabc48cc6, HandlerErrorCode: AccessDenied)
The environment was attempting to modify the RDS database but failed due to an AccessDenied error. It seems that the role aws-elasticbeanstalk-service-role does not have the required permissions to modify the RDS instance.
Additionally, I also noticed this message in the RDS console:
Upgrade required for your databases
You have one or more RDS databases that use RDS Certificate Authorities that have expired or are reaching end-of-life (EOL). After expiry, RDS can no longer accept new connections with these certificates.
I suspect that the expiring RDS Certificate Authorities might also be contributing to the degraded environment health.
How can I resolve the AccessDenied error for the aws-elasticbeanstalk-service-role to ensure it has the appropriate permissions to modify the RDS instance?
Could the expiring RDS Certificate Authorities cause the environment's health to degrade?
Are there any specific steps I need to follow to ensure my environment health recovers once the permissions and certificates are updated?
Any insights or suggestions would be greatly appreciated!
I attempted to restart the app server, but it failed, saying the instance is not in "available." state
I've avoided making any changes to the RDS instance because there is a lot of important data in the database, and I don’t want to risk any potential data loss.
I want to understand the root cause of this issue. The environment has been stable for a year, and this is the first time I’ve encountered any problems. How can I safely debug and resolve these issues without risking data loss? I suspect the expiring RDS certificates could be part of the problem, but I am not certain if they are causing the environment health to degrade.
after a bit of research, i am able to recover my env without rebuilding or cloning it, here are some generic steps to consider
Go to CloudFormation and look for failed stack(for my case Platform updates failed due to RDS permission issue)
Update the required permissions in IAM, go to roles and aws-elasticbeanstalk-service-role, and update the required permissions(RDS permissions in my case)
Back to cloudformation and try to Roll it back into "UPDATE_ROLLBACK_COMPLETE" state now as the permission has been granted now you can try to update it again, in my case it is Updated, and stack status is turned into "UPDATE_COMPLETE"
Now we need AWS support team help to Make our environment available(NOTE: You can contact also support team when stack status is "UPDATE_ROLLBACK_COMPLETE") here are some resources that might help also
https://repost.aws/knowledge-center/cloudformation-update-rollback-failed