Questions tagged [apache-spark]
The apache-spark tag has no usage guidance.
29
questions
0
votes
0
answers
20
views
EOF Exception whnen runnign spark-shell on Hadoop Cluster
The problem
I am trying to install Spark on Hadoop cluster. I've installed and tested Hadoop. I can explore HDFS and run MapReduce example. However, when I tried to install Spark, I can't run it ...
2
votes
1
answer
473
views
Apache Spark Web UI on kubernetes not working as expected
hi im having a problem
i'am deploying Apache spark helm chart on kubernetes
bitnami chart : helm repo add bitnami https://charts.bitnami.com/bitnami
normally the apache spark webui is on port 8080
...
0
votes
0
answers
1k
views
How to read files from a directory having name "/" in S3 bucket?
Code:
val df = spark.read.csv("s3a://sample_bucket//csvFiles/file.csv");
Error:
22/06/23 20:02:57 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system....
1
vote
0
answers
112
views
Throttle Spark Cassandra Connector Reads on a Production Cluster
We're currently running a 24 Node Cassandra Cluster in Production that holds 30Tb of data and handles an average live load of 100k Requests Per Min 24/7. We support multiple partners. One of our ...
1
vote
0
answers
129
views
Fastest way to import files in Spark?
I’m playing around with Spark 3.0.1 and I’m really impressed by the performance with Spark SQL on GB of data.
I’m trying to understand what’s the best way to import multiple JSON files in the Spark ...
0
votes
1
answer
2k
views
Spark-Cassandra-Connector Issue Exception in thread "main" java.lang.NoClassDefFoundError: com/datastax/spark/connector/rdd/reader/RowReaderFactory
What is going wrong with the Spark Cassandra Connector could you please help to solve this?
Scala File:
import com.datastax.spark.connector._
import org.apache.spark.sql.SparkSession
import org.apache....
1
vote
1
answer
1k
views
Zstd parquet decompression
I have parquet file compressed by zstd. It is possible to decompress it somehow? I tried to use zstd command, but without any luck:
[x@xyz tmp]# zstd -d part-00016-303a375a-e443-4f86-a59e-b5d82d15bd26....
1
vote
0
answers
265
views
How to install cosmosdb spark connector in databricks init script
I'm tried to install cosmosdb spark connector (https://docs.microsoft.com/en-us/azure/cosmos-db/spark-connector) in azure databricks on a cluster in init script, but had errors and non working cluster ...
1
vote
1
answer
1k
views
Spark YARN capacity scheduler
I am trying to setup capacity scheduler in Amazon EMR with 2 queues in addition to the default queue. I have successfully created the queues user1 and user2, however when I use spark-submit to run a ...
0
votes
1
answer
228
views
is it possible mix different RHEL OS version in hadoop cluster?
we are using the following HDP cluster with ambari ,
list of nodes and their RHEL version
3 masters machines ( with namenode & resource manager ) , installed on RHEL 7.2
312 DATA-NODES machines ...
0
votes
0
answers
220
views
Any benefits of ZFS over EXT4 for data stream processing on top of HDFS?
I'm working on a data stream processing project in which i will be using Apache Flink and Apache Spark and I want to use HDFS for storage. The development and testing will be done on a single node ...
1
vote
1
answer
164
views
Open a randomized port on a cluster of machines
I'm using Apache Spark, a Java application, to create a cluster of machines. The processes that are launched try to communicate with each other across randomized ports. Is there a way to script the ...
1
vote
0
answers
1k
views
How to use Cassandra with Spark in a Docker image?
(I hope this question is fit for ServerFault, if not, comment and I'll delete it)
I'm trying to create a docker image where Cassandra and Spark would be installed and configured to work together.
I ...
0
votes
1
answer
654
views
Unable to run Spark Cluster on Google DataProc
I am running a 6 node spark cluster on Google Data Proc and within few minutes of launching spark, and performing basic operations, I get the below error
OpenJDK 64-Bit Server VM warning: INFO: os::...
3
votes
0
answers
1k
views
Spark: Pi sample throws SocketTimeoutException in cluster k8s mode
I set up a Spark 2.3.1 cluster on kubernetes, however, I have trouble sending a sample SparkPi job to it:
The submit script I'm using:
bin/spark-submit \
--master k8s://https://10.0.15.7:7077 \
...
1
vote
0
answers
1k
views
Spark: Pi sample throws NoSuchFileException in cluster mode
I set up a Spark 2.3.1 cluster, however, I have trouble sending a sample SparkPi job to it:
Running Spark using the REST application submission protocol.
2018-09-06 13:45:53 INFO ...
1
vote
2
answers
895
views
Equivalent to 'top' command on an EMR cluster?
I have a 3-instance EMR cluster running on AWS, and it's responding very slowly at the moment.
When checking the Hadoop dashboard on port 8088 with my browser, I see "Memory used: 203.5GB", and "...
1
vote
0
answers
299
views
Solr Spark indexing failure due to an error accessing the collection
I am using Solr with Spark in Java to index documents. (Ubuntu 16.0.4)
I have set Zookeeper running on port 2181 and my collection test has two shards
When I launch my code I have a java.lang....
1
vote
0
answers
63
views
spark.dynamicAllocation + setting the spark parameters according to ambari cluster
we want to find the values for the following spark parameters according to inputs as memory on datanode machine , CPU CORE on data node machine , numbers of datanode machine etc ,,
spark....
6
votes
0
answers
10k
views
Spark Error: Failed to Send RPC to Datanode
We have quite few issues with our Spark Thrift server. It is a new Ambari cluster and no Spark jobs are running now.
From the log we can see an error message:
Failed to send RPC 9053901149358924945 ...
1
vote
0
answers
192
views
ambari cluster + when need to set Block replication to 1
We get the following in Spark logs:
java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[...
3
votes
1
answer
4k
views
S3 cross-account bucket permissions
Similar to what is described in this article[0], the company I work for uses a bastion AWS account to store IAM users and other AWS accounts to separate different running environments (prod, dev, etc.)...
0
votes
1
answer
146
views
how to install hadoop2.4.1 in windows with spark 2.0.0
i want to setup a cluster using hadoop in yarn mode..i want to use spark API for map-reduce and will use spark submit to deploy my applications..i want to work on cluster..can anyone help me how to ...
1
vote
1
answer
5k
views
How is the number of RDD partitions decided in Apache Spark?
Question
How is the number of partitions decided by Spark?
Do I need to specify the number of available CPU cores somewhere explicitly so that the number of partitions will be the same (such as ...
3
votes
0
answers
2k
views
Memory problems with Spark application on AWS EMR
I've been trying to get to the bottom of a memory issues for some time now and I simply can't fathom out what the problem is. Any help is greatly appreciated.
The error is:
[![OpenJDK 64-Bit Server ...
6
votes
2
answers
6k
views
How can I run Spark on a cluster using Slurm?
I have written a program example.jar which uses a spark context. How can I run this on a cluster which uses Slurm? This is related to https://stackoverflow.com/questions/29308202/running-spark-on-top-...
5
votes
0
answers
390
views
How many disks for spark_local_dirs?
I'm looking for a solution to improve my Spark cluster performances, I have read from http://spark.apache.org/docs/latest/hardware-provisioning.html:
We recommend having 4-8 disks per node
, I ...
4
votes
1
answer
1k
views
How to configure Spark client running in a Docker container for two-way communication with a remote Spark cluster?
spark-submit seems to require two-way communication with a remote Spark cluster in order to run jobs.
This is easy to configure between machines (10.x.x.x to 10.x.x.x and back) but becomes confusing ...
2
votes
0
answers
748
views
Can Spark be configured to listen on multiple network interfaces/ip addresses?
The nodes in my Spark cluster have two network interfaces each, one public and one private. Using the SPARK_MASTER_IP environment variable, I can configure Spark to listen on port 7077 on one or the ...