Получение ошибки для org.apache.spark.sql.Encoder и поиск отсутствующей или недопустимой зависимости при загрузке файла класса SQLImplicits, LowPrioritySQLImplicits

Я запускаю следующий код для чтения потока kafka с помощью spark-3.2.2 и scala 2.12.0.

Ранее тот же код отлично работал с spark-2.2 и scala 2.11.8,

import spark.implicits._
  val kafkaStream = spark
     .readStream
     .format("kafka")
     .option("kafka.bootstrap.servers", settings.kafka.brokers)
     .option("startingOffsets", "latest")
     .option("failOnDataLoss", "false")
     .option("subscribe", "serviceproblems")
     .load()

  val dataset = kafkaStream.select($"key", $"value").as[(String, String)]
  val mapper = new ObjectMapper
  mapper.registerModule(new ServiceProblemDeserializerModule())

Ошибка при построении кода, как показано ниже

could not find implicit value for evidence parameter of type org.apache.spark.sql.Encoder[(String, String)]
[ERROR]       val dataset = kafkaStream.select($"key", $"value").as[(String, String)]

Есть и некоторые другие ошибки, причину которых я не могу понять, так как доступна ограниченная помощь.

[ERROR] missing or invalid dependency detected while loading class file 'SQLImplicits.class'.
Could not access type Encoder in package org.apache.spark.sql,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'SQLImplicits.class' was compiled against an incompatible version of org.apache.spark.sql.

[ERROR] missing or invalid dependency detected while loading class file 'LowPrioritySQLImplicits.class'.
Could not access type Encoder in package org.apache.spark.sql,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'LowPrioritySQLImplicits.class' was compiled against an incompatible version of org.apache.spark.sql.

[ERROR] missing or invalid dependency detected while loading class file 'package.class'.
Could not access type Row in package org.apache.spark.sql,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'package.class' was compiled against an incompatible version of org.apache.spark.sql.

[ERROR] missing or invalid dependency detected while loading class file 'Dataset.class'.
Could not access type Encoder in package org.apache.spark.sql,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'Dataset.class' was compiled against an incompatible version of org.apache.spark.sql.

Цените любую помощь. Заранее спасибо