Knative configuration as below:
Send 1000 messages to SQS queue, Knative service only creates about 100-150 pods to process all requests.
I checked and found that Kafka broker only sends about 100-180 events to Knative Service at once.
Thus, Knative service only creates about 1xx pods to handle these events.
The remaining events will be failed to send to Knative service and broker will retry sending.
Is there any configuration in Kafka broker to increase number of events that will be sent to Knative service at once?
FYI:
It looks like you're hitting concurrency limits with your Kafka topic and delivery here; in particular, Kafka delivery parallelism (how many events can be delivered at once) is probably bounded by partitions * event-batch-size. It looks like the number of partitions is controlled by the kafka-broker-config ConfigMap and the latter is controlled by max.poll.records in the config-kafka-broker-data-plane ConfigMap.
Another limiting factor of the Kafka implementation is the number of outgoing HTTP requests from a single dispatcher, which appears to be controlled by the maxPoolSize parameter of the data plane.
You didn't see this behavior from the InMemory channel, because it doesn't perform ordered delivery from partitions, and simply attempts to deliver all messages in parallel without tracking order or blocking newer messages on older ones.