0

I just created a cluster using spark-ec2. I followed steps here http://spark.apache.org/docs/latest/ec2-scripts.html .

Now I want to configure all slaves (150), i.e., install Java 8 for both master and all slaves all at once, is there a fast and easy way to have a single terminal for all instances (master and slaves)?

1
  • Google "configuration management system". Ansible may be ideal for this.
    – ceejayoz
    Jun 16, 2016 at 17:55

3 Answers 3

2

You could do this with EC2 Run Command if you have that agent installed, though it may be a better fit for something like a Cloud-Init configuration that you put in the user-data for each instance.

Alternatively consider using AWS OpsWorks to configure the nodes as desired at boot time, or generate a golden image with Packer such that the config is placed at build time instead of boot time.

Configuring 150 nodes via cluster-ssh is a recipe to have inconsistencies in all your nodes, and will be a hassle if one of the nodes needs to be replaced / reprovisioned.

Ansible, Puppet, Salt, CFEngine are other tools that can do this, but not 'interactively'.

2
  • This is a great answer. But the cluster itself is created with a premade Spark script that doesn't allow for fancy preconfigured instances.
    – M-T-A
    Jun 17, 2016 at 15:20
  • @M-T-A Well then modify the script to your liking. You're surely not beholden to having to do everything exactly as the script author intended.
    – EEAA
    Jun 18, 2016 at 1:34
1

In response to your original question, you could install mpssh- Mass Parallel SSH and use it to run commands on all your instances. The first time you run it, you may need to type "yes" 150 times, unless you want to ignore host key checking.

0

Using pssh could be a good solution. Here is an example.

pssh -i -h /root/spark-ec2/slaves dir

The dir command will run on all slaves. pssh comes preinstalled on all cluster.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .