1

I set up a Spark 2.3.1 cluster, however, I have trouble sending a sample SparkPi job to it:

Running Spark using the REST application submission protocol.
2018-09-06 13:45:53 INFO  RestSubmissionClient:54 - Submitting a request to launch an application in spark://10.0.15.7:7077.
2018-09-06 13:46:04 WARN  RestSubmissionClient:66 - Unable to connect to server spark://10.0.15.7:7077.
Warning: Master endpoint spark://10.0.15.7:7077 was not a REST server. Falling back to legacy submission gateway instead.
2018-09-06 13:46:04 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-09-06 13:46:10 ERROR ClientEndpoint:70 - Exception from cluster was: java.nio.file.NoSuchFileException: /opt/spark/examples/jars/spark-examples_2.11-2.3.1.jar
java.nio.file.NoSuchFileException: /opt/spark/examples/jars/spark-examples_2.11-2.3.1.jar
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:526)
    at sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253)
    at java.nio.file.Files.copy(Files.java:1274)
    at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$copyRecursive(Utils.scala:632)
    at org.apache.spark.util.Utils$.copyFile(Utils.scala:603)
    at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:688)
    at org.apache.spark.util.Utils$.fetchFile(Utils.scala:485)
    at org.apache.spark.deploy.worker.DriverRunner.downloadUserJar(DriverRunner.scala:155)
    at org.apache.spark.deploy.worker.DriverRunner.prepareAndRunDriver(DriverRunner.scala:173)
    at org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:92)

The submit script I'm using:

bin/spark-submit \
    --master spark://10.0.15.7:7077 \
    --deploy-mode cluster \
    --name spark-pi \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.kubernetes.container.image=gcr.io/my-project/spark:spark-test \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
    --conf spark.executor.instances=3 \
    /opt/spark/examples/jars/spark-examples_2.11-2.3.1.jar 1000

I built the gcr.io/my-project/spark:spark-test image using default

bin/docker-image-tool.sh -r gcr.io/my-project -t spark-test build push

so /opt/spark/examples/jars/spark-examples_2.11-2.3.1.jar is in the container. Confirmed it:

docker run --rm -it --entrypoint "/bin/ls" gcr.io/my-project/spark:spark-test -l /opt/spark/examples/jars/

My Spark cluster runs on Kubernetes. I'm using spark:// scheme, not new k8s://https:// scheme, so it should work just as regular Spark cluster. IP and ports are visible, including Spark Web UI.

I don't understand where it tries to look for the jar file.

I tried to prepend path with local://, as in spark examples: https://spark.apache.org/docs/2.3.1/running-on-kubernetes.html but it doesn't work with spark:// master url, it throws No FileSystem for scheme: local exception.

RBAC is configured according to the URL above, all pods use spark service account.

I'm out of ideas.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.