4

I made an AWS Beanstalk and RDS instances. I'm testing out a project which syncs FTP and S3 files.

How do I make the typical php artisan queue:work or php artisan queue:listen work in AWS Beanstalk?

I am aware of Redis, ElastiCache, etc. I haven't tested them out. But I'm trying to make it work with the database driver only.

I am also aware of just accessing SSH, but is there any way to process queued jobs without using SSH? Maybe like using .ebconfig?

5

Using the new amazon Linux AMI 2 (recommended)

At the moment the log files are not working for Linux AMI 2, the configuration should be the same whenever this issue is corrected so I left there the commands to create them.

Since the first post the eb roadmap announced that the PHP platform had been updated making things much easier. Below the .ebextensions file there is also a .platform file that needs to be set up. Otherwise nginx will throw 404 errors on all routes.

This image uses Systemd which makes the process much easier an supervisor is not needed anymore. Unfortunately the services keyword is not supported yet for the new images so the service has to be started and restarted using the container_commands keyword.

This file contains all the commands I execute on every production environment, remember to change them to whatever suit your needs:

\.ebextension\01-setup.config
container_commands:
    01-no_dev:
        command: "composer.phar install --optimize-autoloader --no-dev"
    02-config_clear:
        command: "php artisan config:clear"
    03-route_cache:
        command: "php artisan route:cache"
    04-view_cache:
        command: "php artisan view:cache"
    05-migrate: 
        command: "php artisan migrate"
        leader_only: true
    06-queue_start:
        command: "systemctl restart laravel_worker"
files: 
    /opt/elasticbeanstalk/tasks/taillogs.d/laravel-logs.conf: 
        content: /var/app/current/storage/logs/laravel.log
        group: root
        mode: "000755"
        owner: root
    /etc/systemd/system/laraver_worker.service:
        mode: "000755"
        owner: root
        group: root
        content: |
            # Laravel queue worker using systemd
            # ----------------------------------
            #
            # /lib/systemd/system/queue.service
            #
            # run this command to enable service:
            # systemctl enable queue.service

            [Unit]
            Description=Laravel queue worker

            [Service]
            User=nginx
            Group=nginx
            Restart=always
            ExecStart=/usr/bin/nohup /usr/bin/php /var/www/html/laravel-project/artisan queue:work --daemon

            [Install]
            WantedBy=multi-user.target

And the second file:

\.platform\nginx\conf.d\elasticbeanstalk\laravel.conf
location / {
    try_files $uri $uri/ /index.php?$query_string;
    gzip_static on;
}

Using the Using the amazon Linux AMI (previous image)

The best way is to run supervisor to manage the queues under a service to ensure that it is kept running even after a reboot.

In your configuration file inside .ebextensions

1- Install supervisor with the packages. the python keyword uses pip and easy_install under the hood:

packages:
    python:
        supervisor: []

2- Create the supervisor configuration file:

files:
    /usr/local/etc/supervisord.conf:
        mode: "000755"
        owner: root
        group: root
        content: |
            [unix_http_server]
            file=/tmp/supervisor.sock   ; (the path to the socket file)

            [supervisord]
            logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
            logfile_maxbytes=50MB        ; (max main logfile bytes b4 rotation;default 50MB)
            logfile_backups=10           ; (num of main logfile rotation backups;default 10)
            loglevel=info                ; (log level;default info; others: debug,warn,trace)
            pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
            nodaemon=false               ; (start in foreground if true;default false)
            minfds=1024                  ; (min. avail startup file descriptors;default 1024)
            minprocs=200                 ; (min. avail process descriptors;default 200)

            [rpcinterface:supervisor]
            supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

            [supervisorctl]
            serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket

            [include]
            files = /etc/supervisor/conf.d/*.conf

            [inet_http_server]
            port = 127.0.0.1:9001

3- Create the supervisor process file, from the configuring supervisor section in the laravel docs:

files: 
    /etc/supervisor/conf.d/laravel-worker.conf: 
        content: |
            [program:laravel-worker]
            process_name=%(program_name)s_%(process_num)02d
            command=php /var/app/current/artisan queue:work database --sleep=3 --tries=3
            autostart=true
            autorestart=true
            ;user=root
            numprocs=1
            redirect_stderr=true
            ;stdout_logfile=/var/app/current/storage/logs/worker.log
            stopwaitsecs=3600

4- Create a service that runs supervisor. It is the same as this answer with the chkconfig and processname lines added. These will allow us to run it as a service later.

files:
    /etc/init.d/supervisord:
        mode: "000755"
        owner: root
        group: root
        content: |
            #!/bin/bash

            #chkconfig: 345 99 76
            # processname: supervisord

            # Source function library
            . /etc/rc.d/init.d/functions

            # Source system settings
            if [ -f /etc/sysconfig/supervisord ]; then
                . /etc/sysconfig/supervisord
            fi

            # Path to the supervisorctl script, server binary,
            # and short-form for messages.
            supervisorctl=/usr/local/bin/supervisorctl
            supervisord=${SUPERVISORD-/usr/local/bin/supervisord}
            prog=supervisord
            pidfile=${PIDFILE-/tmp/supervisord.pid}
            lockfile=${LOCKFILE-/var/lock/subsys/supervisord}
            STOP_TIMEOUT=${STOP_TIMEOUT-60}
            OPTIONS="${OPTIONS--c /usr/local/etc/supervisord.conf}"
            RETVAL=0

            start() {
                echo -n $"Starting $prog: "
                daemon --pidfile=${pidfile} $supervisord $OPTIONS
                RETVAL=$?
                echo
                if [ $RETVAL -eq 0 ]; then
                    touch ${lockfile}
                    $supervisorctl $OPTIONS status
                fi
                return $RETVAL
            }

            stop() {
                echo -n $"Stopping $prog: "
                killproc -p ${pidfile} -d ${STOP_TIMEOUT} $supervisord
                RETVAL=$?
                echo
                [ $RETVAL -eq 0 ] && rm -rf ${lockfile} ${pidfile}
            }

            reload() {
                echo -n $"Reloading $prog: "
                LSB=1 killproc -p $pidfile $supervisord -HUP
                RETVAL=$?
                echo
                if [ $RETVAL -eq 7 ]; then
                    failure $"$prog reload"
                else
                    $supervisorctl $OPTIONS status
                fi
            }

            restart() {
                stop
                start
            }

            case "$1" in
                start)
                    start
                    ;;
                stop)
                    stop
                    ;;
                status)
                    status -p ${pidfile} $supervisord
                    RETVAL=$?
                    [ $RETVAL -eq 0 ] && $supervisorctl $OPTIONS status
                    ;;
                restart)
                    restart
                    ;;
                condrestart|try-restart)
                    if status -p ${pidfile} $supervisord >&/dev/null; then
                    stop
                    start
                    fi
                    ;;
                force-reload|reload)
                    reload
                    ;;
                *)
                    echo $"Usage: $prog {start|stop|restart|condrestart|try-restart|force-reload|reload}"
                    RETVAL=2
                esac

                exit $RETVAL

5- After all the files are created run the following commands to start the service and to add it so it can be managed:

commands:
  command-1: 
    command: "/etc/init.d/supervisord start"
  command-2:
    command: "chkconfig --add supervisord"

6- Now the services keyword should work, allowing us to set the enabled and ensurerunning flags to true.

services:
    sysvinit:
        supervisord:
            enabled: "true"
            ensureRunning: "true"
            files: 
                - "/usr/local/etc/supervisord.conf"

Place all this in your .config file and deploy to have queues working


Full file - remember to change the chkconfig number and note that I am running a migrate:fresh command

packages:
    python:
        supervisor: []
container_commands:
    01-migrate: 
        command: "php artisan migrate:fresh --seed"
        cwd: /var/app/ondeck
        leader_only: true
files: 
    /opt/elasticbeanstalk/tasks/taillogs.d/laravel-logs.conf: 
        content: /var/app/current/storage/logs/laravel.log
        group: root
        mode: "000755"
        owner: root
    /etc/supervisor/conf.d/laravel-worker.conf: 
        content: |
            [program:laravel-worker]
            process_name=%(program_name)s_%(process_num)02d
            command=php /var/app/current/artisan queue:work database --sleep=3 --tries=3
            autostart=true
            autorestart=true
            ;user=root
            numprocs=1
            redirect_stderr=true
            ;stdout_logfile=/var/app/current/storage/logs/worker.log
            stopwaitsecs=3600
    /usr/local/etc/supervisord.conf:
        mode: "000755"
        owner: root
        group: root
        content: |
            [unix_http_server]
            file=/tmp/supervisor.sock   ; (the path to the socket file)

            [supervisord]
            logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
            logfile_maxbytes=50MB        ; (max main logfile bytes b4 rotation;default 50MB)
            logfile_backups=10           ; (num of main logfile rotation backups;default 10)
            loglevel=info                ; (log level;default info; others: debug,warn,trace)
            pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
            nodaemon=false               ; (start in foreground if true;default false)
            minfds=1024                  ; (min. avail startup file descriptors;default 1024)
            minprocs=200                 ; (min. avail process descriptors;default 200)

            [rpcinterface:supervisor]
            supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

            [supervisorctl]
            serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket

            [include]
            files = /etc/supervisor/conf.d/*.conf

            [inet_http_server]
            port = 127.0.0.1:9001
    /etc/init.d/supervisord:
        mode: "000755"
        owner: root
        group: root
        content: |
            #!/bin/bash

            #chkconfig: <number>
            # processname: supervisord

            # Source function library
            . /etc/rc.d/init.d/functions

            # Source system settings
            if [ -f /etc/sysconfig/supervisord ]; then
                . /etc/sysconfig/supervisord
            fi

            # Path to the supervisorctl script, server binary,
            # and short-form for messages.
            supervisorctl=/usr/local/bin/supervisorctl
            supervisord=${SUPERVISORD-/usr/local/bin/supervisord}
            prog=supervisord
            pidfile=${PIDFILE-/tmp/supervisord.pid}
            lockfile=${LOCKFILE-/var/lock/subsys/supervisord}
            STOP_TIMEOUT=${STOP_TIMEOUT-60}
            OPTIONS="${OPTIONS--c /usr/local/etc/supervisord.conf}"
            RETVAL=0

            start() {
                echo -n $"Starting $prog: "
                daemon --pidfile=${pidfile} $supervisord $OPTIONS
                RETVAL=$?
                echo
                if [ $RETVAL -eq 0 ]; then
                    touch ${lockfile}
                    $supervisorctl $OPTIONS status
                fi
                return $RETVAL
            }

            stop() {
                echo -n $"Stopping $prog: "
                killproc -p ${pidfile} -d ${STOP_TIMEOUT} $supervisord
                RETVAL=$?
                echo
                [ $RETVAL -eq 0 ] && rm -rf ${lockfile} ${pidfile}
            }

            reload() {
                echo -n $"Reloading $prog: "
                LSB=1 killproc -p $pidfile $supervisord -HUP
                RETVAL=$?
                echo
                if [ $RETVAL -eq 7 ]; then
                    failure $"$prog reload"
                else
                    $supervisorctl $OPTIONS status
                fi
            }

            restart() {
                stop
                start
            }

            case "$1" in
                start)
                    start
                    ;;
                stop)
                    stop
                    ;;
                status)
                    status -p ${pidfile} $supervisord
                    RETVAL=$?
                    [ $RETVAL -eq 0 ] && $supervisorctl $OPTIONS status
                    ;;
                restart)
                    restart
                    ;;
                condrestart|try-restart)
                    if status -p ${pidfile} $supervisord >&/dev/null; then
                    stop
                    start
                    fi
                    ;;
                force-reload|reload)
                    reload
                    ;;
                *)
                    echo $"Usage: $prog {start|stop|restart|condrestart|try-restart|force-reload|reload}"
                    RETVAL=2
                esac

                exit $RETVAL
                
commands:
  command-1: 
    command: "/etc/init.d/supervisord start"
  command-2:
    command: "chkconfig --add supervisord"
services:
    sysvinit:
        supervisord:
            enabled: "true"
            ensureRunning: "true"
            files: 
                - "/usr/local/etc/supervisord.conf"
  • i used this. the status shows its running. But the jobs not done. what am i missing. there is no log as well. – romal tandel Apr 24 '20 at 6:14
  • To see where's the error you could connect to the EB instance and check if the service is running, if it's not then something might be wrong with the configuration file. I edited my answer with the full file that's working for me – Aridez May 4 '20 at 18:17
  • i modified the command and it worked.Thanks for this. – romal tandel May 7 '20 at 6:38
  • 1
    @user345602 I'm afraid that this won't restart the queue, although I'm keeping an eye on it since I didn't have the need yet. There's three ways that I may go about that, we can run the "php artisan queue:listen" instead or we can add a container command that should run after the services, so even during the first deploy it should exist. In case it throws an error we could use the "ignoreErrors: true" flag, but it doesn't seem great. I'll update the answer once I know for sure, but these are the three ways to go that I know of. – Aridez Jun 10 '20 at 12:19
  • 1
    Thank you @Aridez. I solved it by creating a new container command "php artisan queue:restart". I'm still testing if this is a reliable solution though. – user345602 Jun 10 '20 at 13:05
0

The ideal way to do this would be to create an .ebextensions/01_queue_worker.config file with the contents that kick off the php artisan queue:work command.

Something like:

container_commands:
  01_queue_worker:
    command: "php artisan queue:work"

Now if you do not want to run the queue worker on the web-server, but only on a separate dedicated worker node, then you can create an environment variable called "WORKER" and set it to true. Then in your ebextensions config file, you can test that variable and only run the script if the "WORKER" variable is "true". That would look something like this:

container_commands:
  01_queue_worker:
    test: '[ "${WORKER}" == "true" ]'
    command: "php artisan queue:work"

As a general rule, any time you need to make any modifications to what is running on Elasticbeanstalk, try looking into ebextensions. This is the mechanism used by AWS to make modifications.

  • The environment variable "WORKER" will have the "true" value across all nodes, won't it? – user345602 Jun 9 '20 at 8:42
-1

I think what you are looking for is this

https://github.com/dusterio/laravel-aws-worker

or

https://lifeofguenter.de/2015/04/27/laravel-queues-with-supervisor-on-elasticbeanstalk

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.