Questions tagged [apache-spark]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
20 views

EOF Exception whnen runnign spark-shell on Hadoop Cluster

The problem I am trying to install Spark on Hadoop cluster. I've installed and tested Hadoop. I can explore HDFS and run MapReduce example. However, when I tried to install Spark, I can't run it ...
Alexey Markov's user avatar
2 votes
1 answer
473 views

Apache Spark Web UI on kubernetes not working as expected

hi im having a problem i'am deploying Apache spark helm chart on kubernetes bitnami chart : helm repo add bitnami https://charts.bitnami.com/bitnami normally the apache spark webui is on port 8080 ...
ossama assaghir's user avatar
0 votes
0 answers
1k views

How to read files from a directory having name "/" in S3 bucket?

Code: val df = spark.read.csv("s3a://sample_bucket//csvFiles/file.csv"); Error: 22/06/23 20:02:57 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system....
Slow_learner's user avatar
1 vote
0 answers
112 views

Throttle Spark Cassandra Connector Reads on a Production Cluster

We're currently running a 24 Node Cassandra Cluster in Production that holds 30Tb of data and handles an average live load of 100k Requests Per Min 24/7. We support multiple partners. One of our ...
Mano's user avatar
  • 11
1 vote
0 answers
129 views

Fastest way to import files in Spark?

I’m playing around with Spark 3.0.1 and I’m really impressed by the performance with Spark SQL on GB of data. I’m trying to understand what’s the best way to import multiple JSON files in the Spark ...
int 2Eh's user avatar
  • 193
0 votes
1 answer
2k views

Spark-Cassandra-Connector Issue Exception in thread "main" java.lang.NoClassDefFoundError: com/datastax/spark/connector/rdd/reader/RowReaderFactory

What is going wrong with the Spark Cassandra Connector could you please help to solve this? Scala File: import com.datastax.spark.connector._ import org.apache.spark.sql.SparkSession import org.apache....
Subhrangshu Adhikary's user avatar
1 vote
1 answer
1k views

Zstd parquet decompression

I have parquet file compressed by zstd. It is possible to decompress it somehow? I tried to use zstd command, but without any luck: [x@xyz tmp]# zstd -d part-00016-303a375a-e443-4f86-a59e-b5d82d15bd26....
Jacfal's user avatar
  • 21
1 vote
0 answers
265 views

How to install cosmosdb spark connector in databricks init script

I'm tried to install cosmosdb spark connector (https://docs.microsoft.com/en-us/azure/cosmos-db/spark-connector) in azure databricks on a cluster in init script, but had errors and non working cluster ...
Роман Коптев's user avatar
1 vote
1 answer
1k views

Spark YARN capacity scheduler

I am trying to setup capacity scheduler in Amazon EMR with 2 queues in addition to the default queue. I have successfully created the queues user1 and user2, however when I use spark-submit to run a ...
sjensen85's user avatar
0 votes
1 answer
228 views

is it possible mix different RHEL OS version in hadoop cluster?

we are using the following HDP cluster with ambari , list of nodes and their RHEL version 3 masters machines ( with namenode & resource manager ) , installed on RHEL 7.2 312 DATA-NODES machines ...
shalom's user avatar
  • 501
0 votes
0 answers
220 views

Any benefits of ZFS over EXT4 for data stream processing on top of HDFS?

I'm working on a data stream processing project in which i will be using Apache Flink and Apache Spark and I want to use HDFS for storage. The development and testing will be done on a single node ...
HUSMEN's user avatar
  • 101
1 vote
1 answer
164 views

Open a randomized port on a cluster of machines

I'm using Apache Spark, a Java application, to create a cluster of machines. The processes that are launched try to communicate with each other across randomized ports. Is there a way to script the ...
activedecay's user avatar
1 vote
0 answers
1k views

How to use Cassandra with Spark in a Docker image?

(I hope this question is fit for ServerFault, if not, comment and I'll delete it) I'm trying to create a docker image where Cassandra and Spark would be installed and configured to work together. I ...
HypeWolf's user avatar
  • 113
0 votes
1 answer
654 views

Unable to run Spark Cluster on Google DataProc

I am running a 6 node spark cluster on Google Data Proc and within few minutes of launching spark, and performing basic operations, I get the below error OpenJDK 64-Bit Server VM warning: INFO: os::...
Tushar Mehta's user avatar
3 votes
0 answers
1k views

Spark: Pi sample throws SocketTimeoutException in cluster k8s mode

I set up a Spark 2.3.1 cluster on kubernetes, however, I have trouble sending a sample SparkPi job to it: The submit script I'm using: bin/spark-submit \ --master k8s://https://10.0.15.7:7077 \ ...
Dzmitry Lazerka's user avatar
1 vote
0 answers
1k views

Spark: Pi sample throws NoSuchFileException in cluster mode

I set up a Spark 2.3.1 cluster, however, I have trouble sending a sample SparkPi job to it: Running Spark using the REST application submission protocol. 2018-09-06 13:45:53 INFO ...
Dzmitry Lazerka's user avatar
1 vote
2 answers
895 views

Equivalent to 'top' command on an EMR cluster?

I have a 3-instance EMR cluster running on AWS, and it's responding very slowly at the moment. When checking the Hadoop dashboard on port 8088 with my browser, I see "Memory used: 203.5GB", and "...
Alexander Engelhardt's user avatar
1 vote
0 answers
299 views

Solr Spark indexing failure due to an error accessing the collection

I am using Solr with Spark in Java to index documents. (Ubuntu 16.0.4) I have set Zookeeper running on port 2181 and my collection test has two shards When I launch my code I have a java.lang....
Dilak's user avatar
  • 11
1 vote
0 answers
63 views

spark.dynamicAllocation + setting the spark parameters according to ambari cluster

we want to find the values for the following spark parameters according to inputs as memory on datanode machine , CPU CORE on data node machine , numbers of datanode machine etc ,, spark....
shalom's user avatar
  • 501
6 votes
0 answers
10k views

Spark Error: Failed to Send RPC to Datanode

We have quite few issues with our Spark Thrift server. It is a new Ambari cluster and no Spark jobs are running now. From the log we can see an error message: Failed to send RPC 9053901149358924945 ...
shalom's user avatar
  • 501
1 vote
0 answers
192 views

ambari cluster + when need to set Block replication to 1

We get the following in Spark logs: java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[...
shalom's user avatar
  • 501
3 votes
1 answer
4k views

S3 cross-account bucket permissions

Similar to what is described in this article[0], the company I work for uses a bastion AWS account to store IAM users and other AWS accounts to separate different running environments (prod, dev, etc.)...
c4urself's user avatar
  • 5,690
0 votes
1 answer
146 views

how to install hadoop2.4.1 in windows with spark 2.0.0

i want to setup a cluster using hadoop in yarn mode..i want to use spark API for map-reduce and will use spark submit to deploy my applications..i want to work on cluster..can anyone help me how to ...
Sadim Nadeem's user avatar
1 vote
1 answer
5k views

How is the number of RDD partitions decided in Apache Spark?

Question How is the number of partitions decided by Spark? Do I need to specify the number of available CPU cores somewhere explicitly so that the number of partitions will be the same (such as ...
mon's user avatar
  • 265
3 votes
0 answers
2k views

Memory problems with Spark application on AWS EMR

I've been trying to get to the bottom of a memory issues for some time now and I simply can't fathom out what the problem is. Any help is greatly appreciated. The error is: [![OpenJDK 64-Bit Server ...
null's user avatar
  • 139
6 votes
2 answers
6k views

How can I run Spark on a cluster using Slurm?

I have written a program example.jar which uses a spark context. How can I run this on a cluster which uses Slurm? This is related to https://stackoverflow.com/questions/29308202/running-spark-on-top-...
mxmlnkn's user avatar
  • 425
5 votes
0 answers
390 views

How many disks for spark_local_dirs?

I'm looking for a solution to improve my Spark cluster performances, I have read from http://spark.apache.org/docs/latest/hardware-provisioning.html: We recommend having 4-8 disks per node , I ...
LucaGuerra's user avatar
4 votes
1 answer
1k views

How to configure Spark client running in a Docker container for two-way communication with a remote Spark cluster?

spark-submit seems to require two-way communication with a remote Spark cluster in order to run jobs. This is easy to configure between machines (10.x.x.x to 10.x.x.x and back) but becomes confusing ...
Leo's user avatar
  • 983
2 votes
0 answers
748 views

Can Spark be configured to listen on multiple network interfaces/ip addresses?

The nodes in my Spark cluster have two network interfaces each, one public and one private. Using the SPARK_MASTER_IP environment variable, I can configure Spark to listen on port 7077 on one or the ...
Leo's user avatar
  • 983