Questions tagged [apache-spark]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
0answers
19 views

Throttle Spark Cassandra Connector Reads on a Production Cluster

We're currently running a 24 Node Cassandra Cluster in Production that holds 30Tb of data and handles an average live load of 100k Requests Per Min 24/7. We support multiple partners. One of our ...
0
votes
0answers
210 views

apache livy session state "starting" and app id "null"

Apache Livy installed on HDP cluster's node which have HDP clients installed. Also the same node have spark configured too. Spark version 2.3.3 Livy version 0.6 and 0.7 (tried both the versions) hdp....
1
vote
0answers
27 views

Fastest way to import files in Spark?

I’m playing around with Spark 3.0.1 and I’m really impressed by the performance with Spark SQL on GB of data. I’m trying to understand what’s the best way to import multiple JSON files in the Spark ...
0
votes
1answer
405 views

Spark-Cassandra-Connector Issue Exception in thread "main" java.lang.NoClassDefFoundError: com/datastax/spark/connector/rdd/reader/RowReaderFactory

What is going wrong with the Spark Cassandra Connector could you please help to solve this? Scala File: import com.datastax.spark.connector._ import org.apache.spark.sql.SparkSession import org.apache....
1
vote
1answer
385 views

Zstd parquet decompression

I have parquet file compressed by zstd. It is possible to decompress it somehow? I tried to use zstd command, but without any luck: [x@xyz tmp]# zstd -d part-00016-303a375a-e443-4f86-a59e-b5d82d15bd26....
0
votes
0answers
154 views

How to install cosmosdb spark connector in databricks init script

I'm tried to install cosmosdb spark connector (https://docs.microsoft.com/en-us/azure/cosmos-db/spark-connector) in azure databricks on a cluster in init script, but had errors and non working cluster ...
1
vote
1answer
721 views

Spark YARN capacity scheduler

I am trying to setup capacity scheduler in Amazon EMR with 2 queues in addition to the default queue. I have successfully created the queues user1 and user2, however when I use spark-submit to run a ...
0
votes
1answer
141 views

is it possible mix different RHEL OS version in hadoop cluster?

we are using the following HDP cluster with ambari , list of nodes and their RHEL version 3 masters machines ( with namenode & resource manager ) , installed on RHEL 7.2 312 DATA-NODES machines ...
0
votes
0answers
113 views

Any benefits of ZFS over EXT4 for data stream processing on top of HDFS?

I'm working on a data stream processing project in which i will be using Apache Flink and Apache Spark and I want to use HDFS for storage. The development and testing will be done on a single node ...
1
vote
1answer
59 views

Open a randomized port on a cluster of machines

I'm using Apache Spark, a Java application, to create a cluster of machines. The processes that are launched try to communicate with each other across randomized ports. Is there a way to script the ...
1
vote
0answers
806 views

How to use Cassandra with Spark in a Docker image?

(I hope this question is fit for ServerFault, if not, comment and I'll delete it) I'm trying to create a docker image where Cassandra and Spark would be installed and configured to work together. I ...
0
votes
1answer
413 views

Unable to run Spark Cluster on Google DataProc

I am running a 6 node spark cluster on Google Data Proc and within few minutes of launching spark, and performing basic operations, I get the below error OpenJDK 64-Bit Server VM warning: INFO: os::...
3
votes
0answers
988 views

Spark: Pi sample throws SocketTimeoutException in cluster k8s mode

I set up a Spark 2.3.1 cluster on kubernetes, however, I have trouble sending a sample SparkPi job to it: The submit script I'm using: bin/spark-submit \ --master k8s://https://10.0.15.7:7077 \ ...
1
vote
0answers
651 views

Spark: Pi sample throws NoSuchFileException in cluster mode

I set up a Spark 2.3.1 cluster, however, I have trouble sending a sample SparkPi job to it: Running Spark using the REST application submission protocol. 2018-09-06 13:45:53 INFO ...
1
vote
2answers
508 views

Equivalent to 'top' command on an EMR cluster?

I have a 3-instance EMR cluster running on AWS, and it's responding very slowly at the moment. When checking the Hadoop dashboard on port 8088 with my browser, I see "Memory used: 203.5GB", and "...
1
vote
0answers
244 views

Solr Spark indexing failure due to an error accessing the collection

I am using Solr with Spark in Java to index documents. (Ubuntu 16.0.4) I have set Zookeeper running on port 2181 and my collection test has two shards When I launch my code I have a java.lang....
1
vote
0answers
53 views

spark.dynamicAllocation + setting the spark parameters according to ambari cluster

we want to find the values for the following spark parameters according to inputs as memory on datanode machine , CPU CORE on data node machine , numbers of datanode machine etc ,, spark....
5
votes
0answers
8k views

Spark Error: Failed to Send RPC to Datanode

We have quite few issues with our Spark Thrift server. It is a new Ambari cluster and no Spark jobs are running now. From the log we can see an error message: Failed to send RPC 9053901149358924945 ...
1
vote
0answers
159 views

ambari cluster + when need to set Block replication to 1

We get the following in Spark logs: java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[...
3
votes
1answer
4k views

S3 cross-account bucket permissions

Similar to what is described in this article[0], the company I work for uses a bastion AWS account to store IAM users and other AWS accounts to separate different running environments (prod, dev, etc.)...
0
votes
1answer
99 views

how to install hadoop2.4.1 in windows with spark 2.0.0

i want to setup a cluster using hadoop in yarn mode..i want to use spark API for map-reduce and will use spark submit to deploy my applications..i want to work on cluster..can anyone help me how to ...
1
vote
1answer
5k views

How is the number of RDD partitions decided in Apache Spark?

Question How is the number of partitions decided by Spark? Do I need to specify the number of available CPU cores somewhere explicitly so that the number of partitions will be the same (such as ...
2
votes
0answers
2k views

Memory problems with Spark application on AWS EMR

I've been trying to get to the bottom of a memory issues for some time now and I simply can't fathom out what the problem is. Any help is greatly appreciated. The error is: [![OpenJDK 64-Bit Server ...
6
votes
2answers
5k views

How can I run Spark on a cluster using Slurm?

I have written a program example.jar which uses a spark context. How can I run this on a cluster which uses Slurm? This is related to https://stackoverflow.com/questions/29308202/running-spark-on-top-...
5
votes
0answers
318 views

How many disks for spark_local_dirs?

I'm looking for a solution to improve my Spark cluster performances, I have read from http://spark.apache.org/docs/latest/hardware-provisioning.html: We recommend having 4-8 disks per node , I ...
4
votes
1answer
993 views

How to configure Spark client running in a Docker container for two-way communication with a remote Spark cluster?

spark-submit seems to require two-way communication with a remote Spark cluster in order to run jobs. This is easy to configure between machines (10.x.x.x to 10.x.x.x and back) but becomes confusing ...
2
votes
0answers
619 views

Can Spark be configured to listen on multiple network interfaces/ip addresses?

The nodes in my Spark cluster have two network interfaces each, one public and one private. Using the SPARK_MASTER_IP environment variable, I can configure Spark to listen on port 7077 on one or the ...