lambda-promtail: ALB connection logs are not properly recognized by the `defaultFilenameRegex` expression

[trc-ikeskin]

Describe the bug

AWS released connection logging for Application Load Balancers alongside their new mTLS functionality: https://aws.amazon.com/blogs/aws/mutual-authentication-for-application-load-balancer-to-reliably-verify-certificate-based-client-identities/ in November 2023.

When one activates those logs they are shipped to the ELB log bucket alongside the already existing access logs. However connection log files have a different file name structure than access logs where they are prefixed with the string "conn_log_".

⚠️ Unfortunately the AWS documentation states the prefix ends with a period, however in reality an underscore is used!

The defaultFilenameRegex does not cover those logs currently which is why below error is thrown:

level=error caller=main.go:269 err="error processing event: type of S3 event could not be determined for object \"AWSLogs/347414xxxxxx/elasticloadbalancing/eu-central-1/2024/02/07/conn_log_347414xxxxxx_elasticloadbalancing_eu-central-1_app.xxxxxxxxx.5ede558f4b0e97a9_20240207T0955Z_10.43.xxx.xxx_b38hdhc6.log.gz\""

https://github.com/grafana/loki/blob/46c6118eb2a2d56406042431dfbed288df6e924b/tools/lambda-promtail/lambda-promtail/s3.go#L78

To Reproduce
Steps to reproduce the behavior:

1. Enable connection logs on an Application Load Balancer.
2. Start scraping ELB logs from the referenced bucket using lambda-promtail.

Expected behavior
Connection log files should be recognized as valid s3_lb log files and handled by the parser. This could be achieve by adjusting the regex like so: AWSLogs\/(?P<account_id>\d+)\/(?P<type>[a-zA-Z0-9_\-]+)\/(?P<region>[\w-]+)\/(?P<year>\d+)\/(?P<month>\d+)\/(?P<day>\d+)\/(?:conn_log_)?\d+\_(?:elasticloadbalancing|vpcflowlogs)\_\w+-\w+-\d_(?:(?P<lb_type>app|net)\.*?)?(?P<src>[a-zA-Z0-9\-]+)

Environment:

• Infrastructure: AWS, AWS Lambda, AWS S3
• Deployment tool: Terraform

Screenshots, Promtail config, or terminal output

level=error caller=main.go:269 err="error processing event: type of S3 event could not be determined for object \"AWSLogs/347414xxxxxx/elasticloadbalancing/eu-central-1/2024/02/07/conn_log_347414xxxxxx_elasticloadbalancing_eu-central-1_app.xxxxxxxxx.5ede558f4b0e97a9_20240207T0955Z_10.43.xxx.xxx_b38hdhc6.log.gz\""

[trc-ikeskin]

On a second thought it might be a good idea to add new log types such as s3_lb_access and s3_lb_connection since the differ a bit in format and further upstream use (e.g. in Grafana) would become easier.

[JEFFCTSAI]

I'm current trying to forward my AWS MSK logs from s3 to Loki using lambda-promtail and also running into this issue

level=error caller=main.go:236 err="error processing event: type of S3 event could not be determined for object \"aws_logs/AWSLogs/<account number>/KafkaBrokerLogs/us-east-1/msk-nonprod-*****/2024-09-30-22/Broker-1_22-05_***.log.gz\""

are we limited to only allow certain types of logs in s3 to be sent to Loki using this lambda?

[msmyth-paramount]

I am running into a similar problem with logs from cloudflare that we are pushing to s3
level=error caller=main.go:236 err="error processing event: type of S3 event could not be determined for object "XYZ/20241030/20241030T173200Z_20241030T173301Z_f564a165.log.gz""

[chuong-dao]

Following this. I also have nginx ingress logs using fluentbit to send to s3. Promtail unable to process it.

"errorMessage":"type of S3 event could not be determined for object
err="error processing event: type of S3 event could not be determined for object

[chaudum]

I'm current trying to forward my AWS MSK logs from s3 to Loki using lambda-promtail and also running into this issue

level=error caller=main.go:236 err="error processing event: type of S3 event could not be determined for object \"aws_logs/AWSLogs/<account number>/KafkaBrokerLogs/us-east-1/msk-nonprod-*****/2024-09-30-22/Broker-1_22-05_***.log.gz\""

are we limited to only allow certain types of logs in s3 to be sent to Loki using this lambda?

@JEFFCTSAI

Yes, there are only 5 types of log supported, see https://github.com/grafana/loki/blob/46c6118eb2a2d56406042431dfbed288df6e924b/tools/lambda-promtail/lambda-promtail/s3.go#L86-L124

[chaudum]

On a second thought it might be a good idea to add new log types such as s3_lb_access and s3_lb_connection since the differ a bit in format and further upstream use (e.g. in Grafana) would become easier.

@trc-ikeskin Any chance you could contribute that change?

[chaudum]

Hey @demon could you open a PR with your commit? I think we need to add a few test cases of different log filenames to verify that it does not break anything.

@trc-ikeskin Could you verify that the change from @demon is working?

[edjshelton]

Rather than requiring log files that are sent to the Lambda function to be parseable by one of a list of regular expressions, it would be very helpful for non-matching log files to be sent anyway as a fallback.

I'm trying to send GitHub audit logs from an S3 bucket to Grafana, and as far as I can tell, using a Lambda function to ship these logs is my only sensible option. But because the S3 object path/name doesn't match any of these regex rules, the Lambda just throws the error mentioned by OP and doesn't send anything.