Questions tagged [apache-spark]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
1answer
35 views

Zstd parquet decompression

I have parquet file compressed by zstd. It is possible to decompress it somehow? I tried to use zstd command, but without any luck: [x@xyz tmp]# zstd -d part-00016-303a375a-e443-4f86-a59e-b5d82d15bd26....
0
votes
0answers
24 views

Push mongodb collections to Amazon redshift

I have got a mongodb database. I need to stream or push data from mongodb to Amazon redshift as and when data arrives in mongodb. Is there any solution for this..? Can this be implemented using Apache ...
0
votes
0answers
78 views

How to install cosmosdb spark connector in databricks init script

I'm tried to install cosmosdb spark connector (https://docs.microsoft.com/en-us/azure/cosmos-db/spark-connector) in azure databricks on a cluster in init script, but had errors and non working cluster ...
0
votes
0answers
60 views

Remove welcome message and banner from spark-shell

Due to the way we design our pipeline, the spark-shell welcome message gets printed out to our Airflow logs, which we'd like to prevent. How do we remove as much of text below upon running of spark-...
0
votes
0answers
130 views

Spark SQL throws an connection exception error?

I have installed Apache Hadoop,spark,hive(meta store as embedded derby) and Scala on a Ubuntu server and configured it individually.Whenever i try to run spark-sql it throws up a error stating 19/...
1
vote
1answer
337 views

Spark YARN capacity scheduler

I am trying to setup capacity scheduler in Amazon EMR with 2 queues in addition to the default queue. I have successfully created the queues user1 and user2, however when I use spark-submit to run a ...
0
votes
1answer
92 views

is it possible mix different RHEL OS version in hadoop cluster?

we are using the following HDP cluster with ambari , list of nodes and their RHEL version 3 masters machines ( with namenode & resource manager ) , installed on RHEL 7.2 312 DATA-NODES machines ...
0
votes
0answers
63 views

Any benefits of ZFS over EXT4 for data stream processing on top of HDFS?

I'm working on a data stream processing project in which i will be using Apache Flink and Apache Spark and I want to use HDFS for storage. The development and testing will be done on a single node ...
1
vote
1answer
51 views

Open a randomized port on a cluster of machines

I'm using Apache Spark, a Java application, to create a cluster of machines. The processes that are launched try to communicate with each other across randomized ports. Is there a way to script the ...
1
vote
0answers
587 views

How to use Cassandra with Spark in a Docker image?

(I hope this question is fit for ServerFault, if not, comment and I'll delete it) I'm trying to create a docker image where Cassandra and Spark would be installed and configured to work together. I ...
0
votes
1answer
334 views

Unable to run Spark Cluster on Google DataProc

I am running a 6 node spark cluster on Google Data Proc and within few minutes of launching spark, and performing basic operations, I get the below error OpenJDK 64-Bit Server VM warning: INFO: os::...
3
votes
0answers
797 views

Spark: Pi sample throws SocketTimeoutException in cluster k8s mode

I set up a Spark 2.3.1 cluster on kubernetes, however, I have trouble sending a sample SparkPi job to it: The submit script I'm using: bin/spark-submit \ --master k8s://https://10.0.15.7:7077 \ ...
1
vote
0answers
482 views

Spark: Pi sample throws NoSuchFileException in cluster mode

I set up a Spark 2.3.1 cluster, however, I have trouble sending a sample SparkPi job to it: Running Spark using the REST application submission protocol. 2018-09-06 13:45:53 INFO ...
1
vote
2answers
261 views

Equivalent to 'top' command on an EMR cluster?

I have a 3-instance EMR cluster running on AWS, and it's responding very slowly at the moment. When checking the Hadoop dashboard on port 8088 with my browser, I see "Memory used: 203.5GB", and "...
1
vote
0answers
219 views

Solr Spark indexing failure due to an error accessing the collection

I am using Solr with Spark in Java to index documents. (Ubuntu 16.0.4) I have set Zookeeper running on port 2181 and my collection test has two shards When I launch my code I have a java.lang....
1
vote
0answers
51 views

spark.dynamicAllocation + setting the spark parameters according to ambari cluster

we want to find the values for the following spark parameters according to inputs as memory on datanode machine , CPU CORE on data node machine , numbers of datanode machine etc ,, spark....
5
votes
0answers
6k views

Spark Error: Failed to Send RPC to Datanode

We have quite few issues with our Spark Thrift server. It is a new Ambari cluster and no Spark jobs are running now. From the log we can see an error message: Failed to send RPC 9053901149358924945 ...
1
vote
0answers
124 views

ambari cluster + when need to set Block replication to 1

We get the following in Spark logs: java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[...
3
votes
1answer
4k views

S3 cross-account bucket permissions

Similar to what is described in this article[0], the company I work for uses a bastion AWS account to store IAM users and other AWS accounts to separate different running environments (prod, dev, etc.)...
0
votes
1answer
50 views

how to install hadoop2.4.1 in windows with spark 2.0.0

i want to setup a cluster using hadoop in yarn mode..i want to use spark API for map-reduce and will use spark submit to deploy my applications..i want to work on cluster..can anyone help me how to ...
1
vote
1answer
4k views

How is the number of RDD partitions decided in Apache Spark?

Question How is the number of partitions decided by Spark? Do I need to specify the number of available CPU cores somewhere explicitly so that the number of partitions will be the same (such as ...
2
votes
0answers
1k views

Memory problems with Spark application on AWS EMR

I've been trying to get to the bottom of a memory issues for some time now and I simply can't fathom out what the problem is. Any help is greatly appreciated. The error is: [![OpenJDK 64-Bit Server ...
6
votes
2answers
4k views

How can I run Spark on a cluster using Slurm?

I have written a program example.jar which uses a spark context. How can I run this on a cluster which uses Slurm? This is related to https://stackoverflow.com/questions/29308202/running-spark-on-top-...
5
votes
0answers
302 views

How many disks for spark_local_dirs?

I'm looking for a solution to improve my Spark cluster performances, I have read from http://spark.apache.org/docs/latest/hardware-provisioning.html: We recommend having 4-8 disks per node , I ...
4
votes
1answer
889 views

How to configure Spark client running in a Docker container for two-way communication with a remote Spark cluster?

spark-submit seems to require two-way communication with a remote Spark cluster in order to run jobs. This is easy to configure between machines (10.x.x.x to 10.x.x.x and back) but becomes confusing ...
2
votes
0answers
538 views

Can Spark be configured to listen on multiple network interfaces/ip addresses?

The nodes in my Spark cluster have two network interfaces each, one public and one private. Using the SPARK_MASTER_IP environment variable, I can configure Spark to listen on port 7077 on one or the ...