Включение SSL на Kafka

Я пытаюсь подключиться к кластеру kafka с SSL, требуемым на брокерах для подключения клиентов. Большинство клиентов могут общаться с брокерами через SSL, поэтому я знаю, что брокеры настроены правильно. Мы намерены использовать двустороннюю SSL-аутентификацию и следовали этим инструкциям: https://docs.confluent.io/current/tutorials/security_tutorial.html#security-tutorial.

Однако у меня есть Java-приложение, которое я хотел бы подключить к брокерам. Я думаю, что квитирование SSL не завершено, и в результате время ожидания запроса к брокеру истекло. То же Java-приложение может без проблем подключаться к брокерам Kafka без поддержки SSL.

Обновлять:

Я столкнулся с этим, когда пытался включить ssl. Во время отладки исключение проверки подлинности включено. Я также вижу, что мое хранилище доверенных сертификатов и хранилище ключей загружены надлежащим образом. Итак, как мне дальше устранять неполадки, связанные с тайм-аутом запроса на обновление метаданных?

org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

Из

private ClusterAndWaitTime waitOnMetadata(String topic, Integer partition, long maxWaitMs) throws InterruptedException {

Когда я запускаю производитель консоли kafka с использованием образа докера bitnami с тем же trustStore / keyStore, переданным как переменные env, он работает нормально.

Это работает:

docker run -it -v /Users/kafka/kafka_2.11-1.0.0/bin/kafka.client.keystore.jks:/tmp/keystore.jks -v /Users/kafka/kafka_2.11-1.0.0/bin/kafka.client.truststore.jks:/tmp/truststore.jks -v /Users/kafka/kafka_2.11-1.0.0/bin/client_ssl.properties:/tmp/client.properties bitnami/kafka:1.0.0-r3 kafka-console-producer.sh --broker-list some-elb.elb.us-west-2.amazonaws.com:9094 --topic test --producer.config /tmp/client.properties

Вот журналы отладки моего клиентского Java-приложения. Будем признательны за любую информацию о том, как решить эту проблему.

                2018-03-13 20:13:38.661  INFO 20653 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8080 (http)
2018-03-13 20:13:38.669  INFO 20653 --- [           main] c.i.aggregate.precompute.Application     : Started Application in 14.066 seconds (JVM running for 15.12)
2018-03-13 20:13:42.225  INFO 20653 --- [           main] o.a.k.clients.producer.ProducerConfig    : ProducerConfig values: 
    acks = all
    batch.size = 16384
    bootstrap.servers = [some-elb.elb.us-west-2.amazonaws.com:9094]
    buffer.memory = 33554432
    client.id = 
    compression.type = lz4
    connections.max.idle.ms = 540000
    enable.idempotence = false
    interceptor.classes = null
    key.serializer = class org.apache.kafka.common.serialization.StringSerializer
    linger.ms = 0
    max.block.ms = 2000
    max.in.flight.requests.per.connection = 5
    max.request.size = 1048576
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
    receive.buffer.bytes = 32768
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 30000
    retries = 2147483647
    retry.backoff.ms = 100
    sasl.jaas.config = null
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.mechanism = GSSAPI
    security.protocol = SSL
    send.buffer.bytes = 131072
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
    ssl.endpoint.identification.algorithm = null
    ssl.key.password = [hidden]
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.location = /Users/kafka/Cluster-Certs/kafka.client.keystore.jks
    ssl.keystore.password = [hidden]
    ssl.keystore.type = JKS
    ssl.protocol = TLS
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.location = /Users/kafka/Cluster-Certs/kafka.client.truststore.jks
    ssl.truststore.password = [hidden]
    ssl.truststore.type = JKS
    transaction.timeout.ms = 60000
    transactional.id = null
    value.serializer = <some class>

2018-03-13 20:13:42.287 TRACE 20653 --- [           main] o.a.k.clients.producer.KafkaProducer     : [Producer clientId=producer-1] Starting the Kafka producer
2018-03-13 20:13:42.841 DEBUG 20653 --- [           main] org.apache.kafka.common.metrics.Metrics  : Added sensor with name bufferpool-wait-time
2018-03-13 20:13:43.062 DEBUG 20653 --- [           main] org.apache.kafka.common.metrics.Metrics  : Added sensor with name buffer-exhausted-records
2018-03-13 20:13:43.217 DEBUG 20653 --- [           main] org.apache.kafka.clients.Metadata        : Updated cluster metadata version 1 to Cluster(id = null, nodes = [some-elb.elb.us-west-2.amazonaws.com:9094 (id: -1 rack: null)], partitions = [])
2018-03-13 20:13:45.670 DEBUG 20653 --- [           main] org.apache.kafka.common.metrics.Metrics  : Added sensor with name produce-throttle-time
2018-03-13 20:13:45.909 DEBUG 20653 --- [           main] org.apache.kafka.common.metrics.Metrics  : Added sensor with name connections-closed:
2018-03-13 20:13:45.923 DEBUG 20653 --- [           main] org.apache.kafka.common.metrics.Metrics  : Added sensor with name connections-created:
2018-03-13 20:13:45.935 DEBUG 20653 --- [           main] org.apache.kafka.common.metrics.Metrics  : Added sensor with name successful-authentication:
2018-03-13 20:13:45.946 DEBUG 20653 --- [           main] org.apache.kafka.common.metrics.Metrics  : Added sensor with name failed-authentication:
2018-03-13 20:13:45.958 DEBUG 20653 --- [           main] org.apache.kafka.common.metrics.Metrics  : Added sensor with name bytes-sent-received:
2018-03-13 20:13:45.968 DEBUG 20653 --- [           main] org.apache.kafka.common.metrics.Metrics  : Added sensor with name bytes-sent:
2018-03-13 20:13:45.990 DEBUG 20653 --- [           main] org.apache.kafka.common.metrics.Metrics  : Added sensor with name bytes-received:
2018-03-13 20:13:46.005 DEBUG 20653 --- [           main] org.apache.kafka.common.metrics.Metrics  : Added sensor with name select-time:
2018-03-13 20:13:46.025 DEBUG 20653 --- [           main] org.apache.kafka.common.metrics.Metrics  : Added sensor with name io-time:
2018-03-13 20:13:46.130 DEBUG 20653 --- [           main] org.apache.kafka.common.metrics.Metrics  : Added sensor with name batch-size
2018-03-13 20:13:46.139 DEBUG 20653 --- [           main] org.apache.kafka.common.metrics.Metrics  : Added sensor with name compression-rate
2018-03-13 20:13:46.147 DEBUG 20653 --- [           main] org.apache.kafka.common.metrics.Metrics  : Added sensor with name queue-time
2018-03-13 20:13:46.156 DEBUG 20653 --- [           main] org.apache.kafka.common.metrics.Metrics  : Added sensor with name request-time
2018-03-13 20:13:46.165 DEBUG 20653 --- [           main] org.apache.kafka.common.metrics.Metrics  : Added sensor with name records-per-request
2018-03-13 20:13:46.179 DEBUG 20653 --- [           main] org.apache.kafka.common.metrics.Metrics  : Added sensor with name record-retries
2018-03-13 20:13:46.189 DEBUG 20653 --- [           main] org.apache.kafka.common.metrics.Metrics  : Added sensor with name errors
2018-03-13 20:13:46.199 DEBUG 20653 --- [           main] org.apache.kafka.common.metrics.Metrics  : Added sensor with name record-size
2018-03-13 20:13:46.250 DEBUG 20653 --- [           main] org.apache.kafka.common.metrics.Metrics  : Added sensor with name batch-split-rate
2018-03-13 20:13:46.275 DEBUG 20653 --- [ad | producer-1] o.a.k.clients.producer.internals.Sender  : [Producer clientId=producer-1] Starting Kafka producer I/O thread.
2018-03-13 20:13:46.329  INFO 20653 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka version : 1.0.0
2018-03-13 20:13:46.333  INFO 20653 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId : aaa7af6d4a11b29d
2018-03-13 20:13:46.369 DEBUG 20653 --- [           main] o.a.k.clients.producer.KafkaProducer     : [Producer clientId=producer-1] Kafka producer started
2018-03-13 20:13:52.982 TRACE 20653 --- [           main] o.a.k.clients.producer.KafkaProducer     : [Producer clientId=producer-1] Requesting metadata update for topic ssl-txn.
2018-03-13 20:13:52.987 TRACE 20653 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Found least loaded node some-elb.elb.us-west-2.amazonaws.com:9094 (id: -1 rack: null)
2018-03-13 20:13:52.987 DEBUG 20653 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Initialize connection to node some-elb.elb.us-west-2.amazonaws.com:9094 (id: -1 rack: null) for sending metadata request
2018-03-13 20:13:52.987 DEBUG 20653 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Initiating connection to node some-elb.elb.us-west-2.amazonaws.com:9094 (id: -1 rack: null)
2018-03-13 20:13:53.217 DEBUG 20653 --- [ad | producer-1] org.apache.kafka.common.metrics.Metrics  : Added sensor with name node--1.bytes-sent
2018-03-13 20:13:53.219 DEBUG 20653 --- [ad | producer-1] org.apache.kafka.common.metrics.Metrics  : Added sensor with name node--1.bytes-received
2018-03-13 20:13:53.219 DEBUG 20653 --- [ad | producer-1] org.apache.kafka.common.metrics.Metrics  : Added sensor with name node--1.latency
2018-03-13 20:13:53.222 DEBUG 20653 --- [ad | producer-1] o.apache.kafka.common.network.Selector   : [Producer clientId=producer-1] Created socket with SO_RCVBUF = 33488, SO_SNDBUF = 131376, SO_TIMEOUT = 0 to node -1
2018-03-13 20:13:53.224 TRACE 20653 --- [ad | producer-1] o.a.k.common.network.SslTransportLayer   : SSLHandshake NEED_WRAP channelId -1, appReadBuffer pos 0, netReadBuffer pos 0, netWriteBuffer pos 0
2018-03-13 20:13:53.224 TRACE 20653 --- [ad | producer-1] o.a.k.common.network.SslTransportLayer   : SSLHandshake handshakeWrap -1
2018-03-13 20:13:53.225 TRACE 20653 --- [ad | producer-1] o.a.k.common.network.SslTransportLayer   : SSLHandshake NEED_WRAP channelId -1, handshakeResult Status = OK HandshakeStatus = NEED_UNWRAP
bytesConsumed = 0 bytesProduced = 326, appReadBuffer pos 0, netReadBuffer pos 0, netWriteBuffer pos 0
2018-03-13 20:13:53.226 TRACE 20653 --- [ad | producer-1] o.a.k.common.network.SslTransportLayer   : SSLHandshake NEED_UNWRAP channelId -1, appReadBuffer pos 0, netReadBuffer pos 0, netWriteBuffer pos 326
2018-03-13 20:13:53.226 TRACE 20653 --- [ad | producer-1] o.a.k.common.network.SslTransportLayer   : SSLHandshake handshakeUnwrap -1
2018-03-13 20:13:53.227 TRACE 20653 --- [ad | producer-1] o.a.k.common.network.SslTransportLayer   : SSLHandshake handshakeUnwrap: handshakeStatus NEED_UNWRAP status BUFFER_UNDERFLOW
2018-03-13 20:13:53.227 TRACE 20653 --- [ad | producer-1] o.a.k.common.network.SslTransportLayer   : SSLHandshake NEED_UNWRAP channelId -1, handshakeResult Status = BUFFER_UNDERFLOW HandshakeStatus = NEED_UNWRAP
bytesConsumed = 0 bytesProduced = 0, appReadBuffer pos 0, netReadBuffer pos 0, netWriteBuffer pos 326
2018-03-13 20:13:53.485 DEBUG 20653 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Completed connection to node -1. Fetching API versions.
2018-03-13 20:13:53.485 TRACE 20653 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Found least loaded node some-elb.elb.us-west-2.amazonaws.com:9094 (id: -1 rack: null)
2018-03-13 20:13:54.992 DEBUG 20653 --- [           main] o.a.k.clients.producer.KafkaProducer     : [Producer clientId=producer-1] Exception occurred during message send:

org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 2000 ms.

2018-03-13 20:13:54.992  INFO 20653 --- [           main] c.i.aggregate.precompute.kafka.Producer  : sent message in callback 
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 2000 ms.
    at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:1124)
    at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:823)
    at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:760)
    at com.intuit.aggregate.precompute.kafka.Producer.send(Producer.java:76)
    at com.intuit.aggregate.precompute.Application.main(Application.java:58)
Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 2000 ms.
Disconnected from the target VM, address: '127.0.0.1:53161', transport: 'socket'