3

I have a Git repository that contains a collection of config files and python scripts.

On the server repository, I have a post-receive hook that will:

  • checkout the master branch,
  • setup a python virtualenv,
  • install the pip requirements,
  • and run some scripts

Like this:

#!/bin/bash
while read oldrev newrev ref; do
    branch=$(git rev-parse --symbolic --abbrev-ref $ref)
    if [[ $branch == "master" ]];
    then
        echo "Ref $ref received in branch $branch"
        echo "setup..."

        cd /srv/repo.git
        git --work-tree=/opt/work-dir checkout master -f

        cd /opt/work-dir
        echo "install virtual environment..." > "setup.log"
        python3 -m virtualenv venv >> "setup.log"
        echo "activate virtual environment..." >> "setup.log"
        source venv/bin/activate >> "setup.log"
        echo "install requirements..." >> "setup.log"
        pip install -r requirements.txt >> "setup.log"
        echo "done." >> "setup.log"

        echo "run scripts..."
        python source/some_script.py --param=value
    else
        echo "Ref $ref received, in branch $branch"
        echo "no action needed."
    fi
done

All of this works fine, but the Git client hangs while the hook is running and will remain like that until I manually cancel. To be clear, the server-side hook does complete successfully, but the client stops logging part way through.

I am aware that I can fix the hanging issue by moving the python setup and execution to a separate bash file and using nohup or disown so that it will continue executing once the terminal closes. However, this would require that I log the result of the script execution separately rather than having it display in the git push log.

  1. What is causing the git client to hang?
  2. How can I monitor the status of the script executions in the git client, without it causing the client to hang?

Calling directly

If I simulate a push by calling the post-receive hook manually, the hook will complete successfully.

hooks/post-receive <<MARK
09998130e13827a097797ff2fd3a973e91665960 fcb0b23a62f47bb73d6ccf2e6bfce324a04eeace master
MARK

Child Processes

I have inspected the child-processes for the hook to see if something isn't exiting properly. As far as I can tell, all child processes exit fully before the parent script continues.

I called ps auxf after every command in the post-receive hook to see what the process tree looked like, and it always returned:

root       711  0.0  0.3  95184  6812 ?        Ss   Feb16   0:00 sshd: username [priv]
username   720  0.0  0.2  95184  4660 ?        S    Feb16   0:04  \_ sshd: username @pts/1,pts/0,pts/2
username   725  0.0  0.1  19892  3712 pts/0    Ss   Feb16   0:00      \_ -bash
root       842  0.0  0.1  49484  3676 pts/0    S    02:49   0:00      |   \_ sudo -i
root       843  0.0  0.1  19940  3768 pts/0    S    02:49   0:00      |       \_ -bash
root      4616  0.0  0.1  50892  3428 pts/0    S    13:59   0:00      |           \_ su - git
git       4617  0.0  0.1  19920  3752 pts/0    S    13:59   0:00      |               \_ -su
git       5108  0.0  0.1  11256  3016 pts/0    S+   17:26   0:00      |                   \_ /bin/bash hooks/post-receive
git       5130  0.0  0.1  38456  3328 pts/0    R+   17:27   0:00      |                       \_ ps auxf

The nest is large because I am logged in over ssh, and sudo'd to root, and then su'd to the git user... but the important part is in the last two lines.

Process of Elimination

I have identified the line that causes the hang:

python3 -m virtualenv venv >> "setup.log"

If I remove this line from the hook, and setup the virtual environment manually before pushing the update, then the push will complete successfully.

  • For the record, I can confirm that running the python setup in a background script stops the client hanging. I have an ok(ish) workaround but I'd appreciate it if someone could tell me what is causing the client to hang. – Marvin Feb 17 at 13:10
2

I believe that either python source/some_script.py --param=value or pip install leaves a hanging child process.

From my experiments, I guess git has some server-side logic to wait for all the children processes left over by the post-receive script. It waits for a hanging child forever.

Please verify your prediction that the post-receive script always completely finishes: run manually post-receive in the shell, then look up the children with ps

  • thanks for the pointers, I've updated my question with the latest. Seems to be something to do with python3 -m virtualenv venv ; works fine when run from bash locally, but causes the git client to hang during a push. – Marvin Feb 17 at 17:47
  • I think ps aufx does not trace properly orphaned grandchilds whose parent have already finished. Look for "strange" processes that also have the same terminal (pts/0 for example). @Marvin – kubanczyk Feb 17 at 18:32
  • It seems that the problem is a timeout issue for the client... if the server-side hook is quiet for more than a few seconds, then the git client concludes "The remote end hung up unexpectedly". I have set more verbose logging messages and it now runs successfully seemingly because it has a more constant stream of messages. – Marvin Feb 17 at 22:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.