0

Attempting to resolve an issue with an upstream AMI having a low max-open ulimit.

I'm able to fill in the limits files, and if I manually disconnect/reconnect the slave, the new limit applies.

I can also put the ulimit changes into cloud-init, but master connects before cloud-init finishes.

The only thing in cloud-init is write a ulimit file. Not sure how to make it any "faster".

User-data:

#!/bin/sh

echo 'ec2-user hard nofile 65535' > /etc/security/limits.d/20-jenkins-maxfiles.conf

Init Script:

#!/bin/bash

echo Cloud-init status:
cloud-init status

echo Setting max-open soft to hard limit.
ulimit -S -n hard

echo Current limits:
ulimit -a

Is there a way to trigger a disconnect/reconnect in ec2-plugin's "Init script" or delay connection to allow cloud-init to finish?

5
  • What AMI are you using specifically? It should honor cloud-init startup. – Oct Nov 29 '18 at 15:54
  • Amazon Linux 2. ami-009d6802948d06e52 which appears to have a default of 4096. Adding cloud-init status to the top of the Init script shows "status: running" in the logs. – rrehbein Nov 29 '18 at 16:00
  • We are using the SpotInstance option, which does effect start-up timing in some cases. It is an issue roughly 4 out of 5 times. – rrehbein Nov 29 '18 at 16:17
  • From what I read on the internet, the cloud-init might be running in // with the systemd startup sequence allowing sshd to start before you have a chance to patch the ulimit file. From there I see only one "reasonable" path: rebuild an AMI that's patched for what you need. Also the ec2 jenkins plugins mentions an init script. It this what's injected in the cloud-config? (read things like: issues.jenkins-ci.org/browse/JENKINS-12771 ) – Oct Nov 29 '18 at 16:22
  • ec2-plugin has both user-data and init-script. The init-script is run at every connection to the slave. user-data per normal ec2. It isn't injecting the init-script into user-data from what I've observed. – rrehbein Nov 29 '18 at 16:26
0

While this does not address the question of having Jenkins reconnect after partially running an initial script, it appears in this context a different cloud-init script type will resolve the issue.

Using Cloud Boothook as the cloud-init script type will have it run prior to SSH starting, and likely many other services systemd starts.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/amazon-linux-ami-basics.html#supported-user-data-formats

https://cloudinit.readthedocs.io/en/latest/topics/format.html#cloud-boothook

Cloud Boothook

  • Begins with #cloud-boothook or Content-Type: text/cloud-boothook.

  • This content is boothook data. It is stored in a file under /var/lib/cloud and then executed immediately.

  • This is the earliest "hook" available. There is no mechanism provided for running it only one time. The boothook must take care of this itself. It is provided with the instance ID in the environment variable INSTANCE_ID. Use this variable to provide a once-per-instance set of boothook data.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.