Как преобразовать строки с помощью Spark в строку с помощью столбца карты

Используйте collect_list и create_map:

df.show()
+----+----+----+----+-----+-----+
|KeyA|KeyB|KeyC|KeyD|Field|Value|
+----+----+----+----+-----+-----+
|   1|   2|   3|   4|    A|    W|
|   1|   2|   3|   4|    A|    X|
|   1|   2|   3|   4|    B|    Y|
|   1|   2|   3|   4|    B|    Z|
|   1|   2|   3|   5|    A|    B|
+----+----+----+----+-----+-----+

df2 = df.groupBy(
    'KEYA','KEYB','KEYC','KEYD','FIELD'
).agg(
    F.collect_list('VALUE').alias('VALUES')
).withColumn(
    'mapped',
    F.create_map('FIELD','VALUES')
).drop('VALUES')

df2.show()
+----+----+----+----+-----+-------------+
|KEYA|KEYB|KEYC|KEYD|FIELD|       mapped|
+----+----+----+----+-----+-------------+
|   1|   2|   3|   4|    A|[A -> [W, X]]|
|   1|   2|   3|   4|    B|[B -> [Y, Z]]|
|   1|   2|   3|   5|    A|   [A -> [B]]|
+----+----+----+----+-----+-------------+

Проверьте код ниже.

scala> df
.select(
    struct(
        $"keya",
        $"keyb",
        $"keyc",
        $"keyd"
    ).as("keys"), // Grouping all keys into struct
    $"field",
    $"value"
)
.groupBy($"field",$"keys") // groupby keys & field
.agg(collect_list($"value").as("value")) // collecting list of values
.select(
    $"keys.*",
    map($"field",$"value").as("map_data") // constructing map.
)
.show(false)

+----+----+----+----+-------------+
|keya|keyb|keyc|keyd|map_data     |
+----+----+----+----+-------------+
|1   |2   |3   |4   |[B -> [Y, Z]]|
|1   |2   |3   |4   |[A -> [W, X]]|
|1   |2   |3   |5   |[A -> [B]]   |
+----+----+----+----+-------------+