aws-lambda

Cloudwatch logs filter to trigger lambda then extract values from log data


Please I have got a question from the subject-line. I want to create a AWS CloudWatch log or Event to trigger Lambda function from filter pattern then extract values from that log data as output to lambda function in python. Example: Filter name: abcd value to extract: 01234 to the lambda function. from log data

log Data: abcd:01234 Any ideas?


Solution

  • Here is a simple way to capture the events from CloudWatch. The log data is in the message. You could process here or send it on to Firehose and transform there. Alternatively, you could send CloudWatch directly to Firehose with a subscription but I think that haas to be done with the AWS CLI.

    import boto3
    import gzip
    import json
    import base64
    
    firehose = boto3.client('firehose',region_name='us-east-2')
    
    def print_result(firehose_return):
        records_error = int(firehose_return['FailedPutCount'])
        records_sent = len(firehose_return['RequestResponses'])
        return 'Firehose sent %d records, %d error(s)' % (records_sent,records_error )
    
    def lambda_handler(events, context):
    
        cw_encoded_logs_data = events['awslogs']['data']
        compressed_payload = base64.b64decode(cw_encoded_logs_data)
        cw_decoded_logs_data = gzip.decompress(compressed_payload)
        cw_all_events = json.loads(cw_decoded_logs_data)
    
        records = []
    
        for event in cw_all_events['logEvents']:
            log_event = {
                "Data": str(event['message']) + '\n'
                }
            records.insert(len(records),log_event)
        
            if len(records) > 499:
                firehose_return = firehose.put_record_batch(
                    DeliveryStreamName = 'streamname   ',
                    Records = records
                )
                print_result(firehose_return)
                records = []
    
        if len(records) > 0:
            firehose_return = firehose.put_record_batch(
                DeliveryStreamName = 'streamname',
                Records = records
    )
    
        print(print_result(firehose_return))