Embedded Kafka doesn't work under certain network conditions

[glorat]

In particular if my laptop is not connected to any network, I get LEADER_NOT_AVAILABLE errors that don't go away. I can reproduce this issue by running the test suite with my laptop offline from any network. The issue seems to be https://stackoverflow.com/questions/35788697/leader-not-available-kafka-in-console-producer

I'll provide a proposed patch...

[glorat]

I have a workaround pending a patch!

val kafkaServer = new KafkaServer(kafkaConfig = KafkaServer.defaultConfig + (KafkaConfig.AdvertisedHostNameProp -> "localhost"))

[david-mcgillicuddy-ovo]

Unfortunately this workaround doesn't work for me - I'm using the massively cargo-culted config which is basically a collection of all stack overflow answers on the topic:

  private val port = 9345
  private val kafkaServer = new KafkaServer(
    kafkaPort = port,
    kafkaConfig = KafkaServer.defaultConfig ++ Map(
      KafkaConfig.AdvertisedHostNameProp -> "localhost",
      KafkaConfig.AdvertisedPortProp -> port.toString,
      KafkaConfig.ListenersProp -> s"PLAINTEXT://0.0.0.0:$port",
      KafkaConfig.PortProp -> port.toString,
      KafkaConfig.AutoCreateTopicsEnableProp -> "true"
    )
  )

But still get

382 [default-akka.kafka.default-dispatcher-8] WARN  o.a.k.c.NetworkClient - [Consumer clientId=consumer-1, groupId=foo] Error while fetching metadata with correlation id 2 : {test=LEADER_NOT_AVAILABLE}
3488 [default-akka.kafka.default-dispatcher-8] WARN  o.a.k.c.NetworkClient - [Consumer clientId=consumer-1, groupId=foo] Error while fetching metadata with correlation id 4 : {test=LEADER_NOT_AVAILABLE}
5570 [default-akka.kafka.default-dispatcher-8] WARN  o.a.k.c.NetworkClient - [Consumer clientId=consumer-1, groupId=foo] Connection to node 1 could not be established. Broker may not be available.
5678 [default-akka.kafka.default-dispatcher-8] WARN  o.a.k.c.NetworkClient - [Consumer clientId=consumer-1, groupId=foo] Connection to node 1 could not be established. Broker may not be available.

I've tried putting similar settings in my consumer config but simply get warnings like

3274 [default-akka.kafka.default-dispatcher-6] WARN  o.a.k.c.c.ConsumerConfig - The configuration 'advertised.host.name' was supplied but isn't a known config.

Any idea? I'm on OSX El Capitan if that makes a difference. For reference running the kafka quickstart scripts with their default configs works perfectly with the same consumer settings.

[glorat]

I'm on OSX too, albeit Sierra and it worked for me so who knows...

Given it is saying it doesn't recognise the config key, maybe you don't have a sufficiently recent version of Kafka. At the time, I was on 1.0.0