We just upgraded from CDH 5.3.6 to 5.10.0, and started getting errors when trying to write to Kafka topics. We have the default settings on everything, no SSL or Kerberos authentication enabled. When use the console producer to write to one of my topics, I get this error:
/usr/bin/kafka-console-producer --broker-list=myhost1.dev.com:9092,myhost2.dev.com:9092 --topic test
17/03/06 21:00:57 INFO utils.AppInfoParser: Kafka version : 0.10.0-kafka-2.1.0
17/03/06 21:00:57 INFO utils.AppInfoParser: Kafka commitId : unknown
x
17/03/06 21:00:59 WARN clients.NetworkClient: Error while fetching metadata with correlation id 0 : {test=TOPIC_AUTHORIZATION_FAILED}
Looking at /var/log/kafka/, I see a bunch of these exceptions:
2017-03-06 21:00:26,964 WARN org.apache.sentry.provider.common.HadoopGroupMappingService: Unable to obtain groups for ANONYMOUS
java.io.IOException: No groups found for user ANONYMOUS
at org.apache.hadoop.security.Groups.noGroupsForUser(Groups.java:190)
at org.apache.hadoop.security.Groups.getGroups(Groups.java:210)
at org.apache.sentry.provider.common.HadoopGroupMappingService.getGroups(HadoopGroupMappingService.java:60)
at org.apache.sentry.provider.common.ResourceAuthorizationProvider.getGroups(ResourceAuthorizationProvider.java:167)
at org.apache.sentry.provider.common.ResourceAuthorizationProvider.doHasAccess(ResourceAuthorizationProvider.java:97)
at org.apache.sentry.provider.common.ResourceAuthorizationProvider.hasAccess(ResourceAuthorizationProvider.java:91)
at org.apache.sentry.kafka.binding.KafkaAuthBinding.authorize(KafkaAuthBinding.java:212)
at org.apache.sentry.kafka.authorizer.SentryKafkaAuthorizer.authorize(SentryKafkaAuthorizer.java:63)
at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$authorize$2.apply(KafkaApis.scala:321)
at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$authorize$2.apply(KafkaApis.scala:321)
at scala.Option.map(Option.scala:146)
at kafka.server.KafkaApis.kafka$server$KafkaApis$$authorize(KafkaApis.scala:321)
at kafka.server.KafkaApis$$anonfun$30.apply(KafkaApis.scala:702)
at kafka.server.KafkaApis$$anonfun$30.apply(KafkaApis.scala:702)
at scala.collection.TraversableLike$$anonfun$partition$1.apply(TraversableLike.scala:314)
at scala.collection.TraversableLike$$anonfun$partition$1.apply(TraversableLike.scala:314)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:94)
at scala.collection.TraversableLike$class.partition(TraversableLike.scala:314)
at scala.collection.AbstractTraversable.partition(Traversable.scala:104)
at kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:702)
at kafka.server.KafkaApis.handle(KafkaApis.scala:79)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
I've been looking for a solution to this, but have come up empty so far. Do I need to assign the ANONYMOUS user to some groups somewhere? I was able to write messages to my topics in CDH 5.3.6, but it appears something has gone wrong in the upgrade.
Just trying to get the helloWorld/Quickstart example to work again on our DEV Kafka after upgrading to CDH 5.10.0.
----------------- Temporary workaround solution ---
In cloudera manager 5.10 there is a super.users property in the kafka configuration. Adding ANONYMOUS to that list, allowed me to produce and consume from my topics.
I had already tried doing this in /opt/cloudera/parcels/KAFKA-2.1.0-1.2.1.0.p0.115/etc/kafka/conf.dist/server.properties, which had no effect. So Cloudera must be managing these values elsewhere.
Kafka strictly distinguishes between authentication and authorization - even if you have authentication via Kerb or SSL turned of it is still possible to turn authorization on via the following parameter:
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
This will make Kafka check ACLs for every access - since authentication is turned of in your case though, every user will be evaluated as ANONYMOUS and denied, if there are no ACLs set for this user.
You can delete that setting from your config which should make Kafka return to its old, trusting, self. I am not sure where you'd do this in Cloudera Manager though, so an alternative would be to add ANONYMOUS to the list of super users which is available in CM. Or of course just define an ACL to allow access to ANONYMOUS.
For production use later on you should probably set up either SSL or Kerberos and define proper ACLs if there is any chance of the cluster being accessed from the outside.