Я следовал документации kafka https://kafka.apache.org/documentation/#security_sasl_config и мог запустить kafka в режиме SASL_PLAINTEXT, затем я захотел использовать kafka в режиме SASL_SSL, поэтому я следовал документации https://kafka.apache.org/documentation/#security_ssl для настройки SSL, возможно, есть некоторые проблемы в документации, но я наконец решил их, сгенерировал сертификаты и сохранил их в магазине. Шаги примерно такие:

1. create keystore
keytool -keystore server.keystore.jks -alias localhost -validity 365 -genkey -keyalg RSA -storetype pkcs12

2. generate certificate-signing-requests (CSR)
# no parameter -destkeystoretype for keytool
# keytool -keystore server.keystore.jks -alias localhost -validity 365 -genkey -keyalg RSA -destkeystoretype pkcs12 -ext SAN=DNS:localhost,IP:12.12.36.25
keytool -keystore server.keystore.jks -alias localhost -validity 365         -keyalg RSA                          -certreq -file server.csr -storetype pkcs12

3. create serial.txt, index.txt and openssl-ca.cnf file 
echo 01 > serial.txt    
touch index.txt

4. generate your CA (Certificate Authority)
# there is no openssl command for windows, you have to run it in the 'git bash' 
openssl req -x509 -config openssl-ca.cnf -newkey rsa:4096 -sha256 -nodes -out cacert.pem -outform PEM

5. add the generated CA to the **clients' truststore** so that the clients can trust this CA, also add it to server truststore.
keytool -keystore client.truststore.jks -alias CARoot -import -file cacert.pem
keytool -keystore server.truststore.jks -alias CARoot -import -file cacert.pem

6. Signing your CA
openssl ca -config openssl-ca.cnf -policy signing_policy -extensions signing_req -out server.cert -infiles server.csr

7. Import both the certificate of the CA and the signed certificate into the keystore:
keytool -keystore server.keystore.jks -alias CARoot -import -file cacert.pem
keytool -keystore server.keystore.jks -alias localhost -import -file server.cert 

Затем я изменил файл server.properties.

listeners=SASL_SSL://localhost:9094
security.inter.broker.protocol=SASL_SSL
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN

# ssl configurations
ssl.keystore.location=/sdk/kafka_2.13-3.7.0/ssl_certs/server.keystore.jks
ssl.keystore.password=******
ssl.key.password=******
ssl.truststore.location=/sdk/kafka_2.13-3.7.0/ssl_certs/server.truststore.jks
ssl.truststore.password=******
ssl.client.auth=required
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1

Zookeeper запустил без проблем, при запуске сервера Kafka вылезла следующая ошибка, зациклился :-(

[2024-06-06 09:56:13,154] INFO [Controller id=0, targetBrokerId=0] Node 0 disconnected. (org.apache.kafka.clients.NetworkClient)
[2024-06-06 09:56:13,154] ERROR [Controller id=0, targetBrokerId=0] Connection to node 0 (localhost/127.0.0.1:9094) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2024-06-06 09:56:13,154] INFO [Controller id=0, targetBrokerId=0] Client requested connection close from node 0 (org.apache.kafka.clients.NetworkClient)
[2024-06-06 09:56:13,273] INFO [Controller id=0, targetBrokerId=0] Failed authentication with localhost/127.0.0.1 (channelId=0) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2024-06-06 09:56:13,273] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /127.0.0.1 (channelId=127.0.0.1:9094-127.0.0.1:54581-56) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2024-06-06 09:56:13,276] INFO [Controller id=0, targetBrokerId=0] Node 0 disconnected. (org.apache.kafka.clients.NetworkClient)
[2024-06-06 09:56:13,276] ERROR [Controller id=0, targetBrokerId=0] Connection to node 0 (localhost/127.0.0.1:9094) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2024-06-06 09:56:13,276] INFO [Controller id=0, targetBrokerId=0] Client requested connection close from node 0 (org.apache.kafka.clients.NetworkClient)
[2024-06-06 09:56:13,383] INFO [Controller id=0, targetBrokerId=0] Failed authentication with localhost/127.0.0.1 (channelId=0) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2024-06-06 09:56:13,383] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /127.0.0.1 (channelId=127.0.0.1:9094-127.0.0.1:54582-56) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2024-06-06 09:56:13,383] INFO [Controller id=0, targetBrokerId=0] Node 0 disconnected. (org.apache.kafka.clients.NetworkClient)
[2024-06-06 09:56:13,383] ERROR [Controller id=0, targetBrokerId=0] Connection to node 0 (localhost/127.0.0.1:9094) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2024-06-06 09:56:13,384] INFO [Controller id=0, targetBrokerId=0] Client requested connection close from node 0 (org.apache.kafka.clients.NetworkClient)
[2024-06-06 09:56:13,493] INFO [Controller id=0, targetBrokerId=0] Failed authentication with localhost/127.0.0.1 (channelId=0) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2024-06-06 09:56:13,493] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /127.0.0.1 (channelId=127.0.0.1:9094-127.0.0.1:54583-56) (SSL handshake failed) (org.apache.kafka.common.network.Selector)

Я ожидал, что сервер Kafka сможет запуститься с включенным SSL.