для запуска команды spark-submit из EC2 я генерирую токен для пользователя с
https://awscli.amazonaws.com/v2/documentation/api/latest/reference/eks/get-token.html
aws eks get-token --cluster-name spark-on-eks --region eu-west-2 > token
Это возвращается с токеном
{
"kind": "ExecCredential",
"apiVersion": "client.authentication.k8s.io/v1beta1",
"spec": {},
"status": {
"expirationTimestamp": "2023-03-30T10:02:17Z",
"token": "k8s-aws-v1.aHR0cHM6Ly9zdHMuZXUtd2VzdC0yLmFtYXpvbmF3cy5jb20vP0FjdGlvbj1HZXRDYWxsZXJJZGVudGl0eSZWZXJzaW9uPTIwMTEtMDYtMTUmWC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBUldRUzZVQ0JKTjdWQjZFQiUyRjIwMjMwMzMwJTJGZXUtd2VzdC0yJTJGc3RzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyMzAzMzBUMDk0ODE3WiZYLUFtei1FeHBpcmVzPTYwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCUzQngtazhzLWF3cy1pZCZYLUFtei1TaWduYXR1cmU9NGUxZGE5MzkxYjZmMGFhM2M3ZDQyM2EyMGQyNTRhMWU4NWQ4M2IwNmEzNTRiNmExOWRkMWIxNmMzODZiYTA0YQ"
}
}
В spark-submit я передаю это как
export K8S_TOKEN=/var/run/secrets/kubernetes.io/serviceaccount/token
--conf spark.kubernetes.authenticate.submission.oauthTokenFile=$K8S_TOKEN
но выдает эту ошибку
Exception in thread "main" java.lang.IllegalArgumentException: Unexpected char 0x0a at 8 in Authorization value: Bearer {
"kind": "ExecCredential",
"apiVersion": "client.authentication.k8s.io/v1beta1",
"spec": {},
"status": {
"expirationTimestamp": "2023-03-30T10:02:17Z",
"token": "k8s-aws-v1.aHR0cHM6Ly9zdHMuZXUtd2VzdC0yLmFtYXpvbmF3cy5jb20vP0FjdGlvbj1HZXRDYWxsZXJJZGVudGl0eSZWZXJzaW9uPTIwMTEtMDYtMTUmWC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBUldRUzZVQ0JKTjdWQjZFQiUyRjIwMjMwMzMwJTJGZXUtd2VzdC0yJTJGc3RzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyMzAzMzBUMDk0ODE3WiZYLUFtei1FeHBpcmVzPTYwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCUzQngtazhzLWF3cy1pZCZYLUFtei1TaWduYXR1cmU9NGUxZGE5MzkxYjZmMGFhM2M3ZDQyM2EyMGQyNTRhMWU4NWQ4M2IwNmEzNTRiNmExOWRkMWIxNmMzODZiYTA0YQ"
}
}
at okhttp3.Headers.checkValue(Headers.java:272)
at okhttp3.Headers$Builder.add(Headers.java:312)
at okhttp3.Request$Builder.addHeader(Request.java:196)
at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createHttpClient$3(HttpClientUtils.java:150)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
at okhttp3.RealCall.execute(RealCall.java:93)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:490)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:451)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:252)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:879)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:341)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:84)
at org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:139)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$3(KubernetesClientApplication.scala:213)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$3$adapted(KubernetesClientApplication.scala:207)
at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2611)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:207)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:179)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
23/03
Является ли это недопустимым токеном в данном случае или есть проблема с синтаксисом сгенерированного токена?
Спасибо
Решение состояло в том, чтобы избавиться от символа новой строки из сгенерированного токена.
aws eks get-token --cluster-name $CLUSTER_NAME --region $REGION --output text | base64 | tr -d '\n' > token