When I use CloudWatch > Metrics > All metrics > Browse > Logs > Account Metrics > ErrorCount, I see events that were produced outside of anything I can control. I am sure, since this is in a sandbox environment with zero activity in my Lambdas. In my log groups, there are only lambdas, access logs, and insights. Zero activity everywhere.
Moreover, I have an alarm for ErrorCount, and when it is active, I can go to "CloudWatch Logs" automatic dashboard and see the error event here when I enable the "In alarm" card, but only when there is no filter by resource group. When I select my resource group, the event disappears.
My question is: what are those events, and where can I check them? Any information or theories are welcome.
Here's the description of the ErrorCount metrics of `AWS/Logs` from the documentation:
The number of API operations performed in your account that resulted in errors.
If you see a non-zero number that means that there was an API call for the logs service that resulted in an error.
This is not necessary a bad thing, in some flows errors can be expected and handled accordingly. For example a Lambda function when it runs the first time, it attempts to create a log stream but the log group isn't created yet. When it gets the error, then it retries with creating the log group.
If this API call is supported by Cloutrail you could see it there:
If you couldn't find anything that's failing then check the minute before and after. If you still couldn't find the API that would mean the API isn't published by the Cloudtrail. However you could still get get some hints on what was happening in your account in that couple of minutes by removing all the filters.
If this API call is published by your code then it would be in one of the Lambdas. However if the Lambda function themselves don't have any code that directly interface with Cloudwatch logs service, then it must be something AWS does on your behalf and so it would their responsibility to monitor and fix whatever it is.
Of course, last but not the least, if you have developer support plan or higher, you could ask AWS Support if this is an issue for your account.