Questions tagged [amazon-emr]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
1 answer
3k views

my emr cluster is being terminated with error after the status is being set to starting

Hi when I create EMR cluster. The status says it is being created but after 58 minutes it throws in error saying Master - 1: Error provisioning instances. Error message(Screenshot of error attached) I ...
rgzv's user avatar
  • 3
0 votes
0 answers
110 views

AWS EC2 with and without EMR and Spark will not SSH to Port 22

I'm on a MacBookPro. I tried starting an EC2 instance with EMR and Rstudio and Spark. I got a port 22 time out. I asked AWS for help and took down my firewall and restarted my modem. Still, nothing ...
Ryan's user avatar
  • 101
0 votes
1 answer
684 views

installing packages in AWS EMR

I'm trying to install Google Tink in AWS EMR 5.28.0 without much luck. It looks like the AWS EMR image is rather strange in nature. Any ideas? sc.install_pypi_package("tink") error: Could ...
Koenig Lear's user avatar
1 vote
0 answers
129 views

Fastest way to import files in Spark?

I’m playing around with Spark 3.0.1 and I’m really impressed by the performance with Spark SQL on GB of data. I’m trying to understand what’s the best way to import multiple JSON files in the Spark ...
int 2Eh's user avatar
  • 193
0 votes
1 answer
1k views

AWS FSx for lustre with S3 vs EMR (with EMRFS) for spark jobs

We are currently using EMR for easy job submission for our spark jobs. Recently I came across the "FSx lustre + S3" solution that is being advertised as ideal for HPC situations. EMRFS however is also ...
dimisjim's user avatar
  • 245
1 vote
1 answer
1k views

Spark YARN capacity scheduler

I am trying to setup capacity scheduler in Amazon EMR with 2 queues in addition to the default queue. I have successfully created the queues user1 and user2, however when I use spark-submit to run a ...
sjensen85's user avatar
0 votes
1 answer
699 views

psql installation on linux requires systemd

I am installing psql on my AWS EMR (EC2 instance) which is Amazon Linux (not Amazon Linux 2). I am getting an error after running the command sudo yum install -y postgresql10 Error: Package: ...
Muhammad Imran Tariq's user avatar
2 votes
1 answer
2k views

AWS S3 costs for when AWS EMR uses it

When I run an AWS EMR cluster and it reads from and writes to an AWS S3 bucket (or multiple buckets), what are the costs for that data transfer? Is that data transfer? Free, because it's ...
Marco's user avatar
  • 123