I have localstack running a mock Kinesis stream within my Docker Compose at http://localstack:4645
I am using the Flink Kinesis ingress io type to attempt to connect to the stream.
https://nightlies.apache.org/flink/flink-statefun-docs-release-3.2/docs/modules/io/aws-kinesis/
Following the above I created
kind: io.statefun.kinesis.v1/ingress
spec:
id: com/sentiment
awsRegion:
type: specific
id: us-east-1
awsCredentials:
type: custom-endpoint
endpoint: https://localhost:4566
id: us-east-1
accessKeyId: key
secretAccessKey: secret
startupPosition:
type: latest
streams:
- stream: customer-details
valueType: namespace/SentimentEvent
targets:
- namespace/sentiment
clientConfigProperties:
- SocketTimeout: 9999
- MaxConnections: 15
I then start up my Docker Compose file which contains the Flink StateFun image but I get this error within my logs:
Caused by: org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException: Invalid AWS credential type: custom-endpoint; valid values are [default, basic, profile] (through reference chain: org.apache.flink.statefun.flink.io.kinesis.binders.ingress.v1.RoutableKinesisIngressSpec$Builder["awsCredentials"])
It's confusing as the docs say:
Custom Endpoint #
Connects to an AWS region through a non-standard AWS service endpoint. This is typically used only for development and testing purposes.
awsCredentials:
type: custom-endpoint
endpoint: https://localhost:4567
id: us-west-1
But it doesn't seem to be valid configuration.
I can access the Localstack stream fine with the awslocal
tool and my other applications within my Docker Compose so I know that's not the issue.
I can't see seem to figure out the issue.
Turns out endpoint needs to go under the awsRegion section in the config.
awsRegion:
type: custom-endpoint
id: us-east-1
endpoint: https://localstack:4566