I'm trying to capture event-stream data on my website which doesn't get a ton of traffic -- maybe around 50-200 DAU -- but I'd like to be set up to handle more, if necessary.
Since there isn't a ton of traffic, I figured something like Kinesis might be overkill. My site currently uses API Gateway + Lambda to handle some server-side logic and I figured within each Lambda handler function, I could push event data to SQS, and then schedule a Lambda function to periodically pull a batch of data from SQS, group it together and store it in S3.
Would my approach work / be reasonable? Would you suggest using something like Kinesis, RabbitMQ, or Kafka instead? or is there another approach you'd take?
Thanks for your help!
Any sort of streaming data is tailor made for Kinesis. And if you end up gaining more traffic at some point, by using Kinesis now you'll be ready to scale.
I'd figure out the lowest cost of Kinesis shard you can get and go with that.
Pricing
Shard Hour (1MB/second ingress, 2MB/second egress) $0.015
PUT Payload Units, per 1,000,000 units $0.014
A penny per hour and a penny for 1 million puts might cover your existing need.