Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

By default, I get an error that I have too many open files from the process. If I lift the limit manually, I get an error that I'm out of memory. For whatever reason, it seems that Git Annex in its current state is not optimised for this sort of task (adding thousands of files to a repository at once).

As a possible solution, my next thought was to do something like:

cd /
find . -type d | xargs git annex add --$NONRECURSIVELY
find . -type f | xargs git annex add
    # Need to add parent directories of each file first or adding files fails

The problem with this solution is that there doesn't seem from the documentation to be a way to non-recursively add a directory in Git Annex. Is there something I'm missing or a workaround for this?

If my proposed solution is a dead end, are there other ways that people have solved this problem?

share|improve this question

2 Answers 2

up vote 2 down vote accepted

Small update:

  1. This entire problem was totally user error. My root drive is a 256 GB SSD, while one of the folders I was attempting to add mapped to a 1.5 TB RAID 1 array. No matter how I tried to accomplished this, it would have inevitably tried to copy more files into the /.git folder than the drive could have fit (duh). No idea what I thought was going to happen :/.

  2. This is why you don't mess with system directories...

  3. Initialised the Git Annex repository on the 1.5 TB drive instead and just copied the root-level directories I wanted backed up. The normal git annex add . worked brilliantly, and my repository has been in the process of backing itself up to Glacier for the past five days or so using these Annex-Glacier hooks with little issue.

share|improve this answer

Update: do not do this.

Evidently, Git Annex moves each file added to a repository to some directory structure in .git/annex/objects, then replaces them with symlinks to the real files in .git. This would be fine if I hadn't first experimented with adding /etc.

Needless to say, server hosed. Luckily, I came up with a fix:

find /etc -type l | while read file ; do realpath=`realpath "${file}"` ; rm "${file}" ; cp -rfa "${realpath}" "${file}" ; done

Edit: Disregard; I'm stupid; system's still hosed; this is going to be a long night.

Second edit: Managed to unhose system . It involved a lot of manual reconstruction of /etc and reinstalling every package, including reconfiguring/unbreaking a large number of them and debugging/solving a ton of problems with APT. Would not try this again.


As far as the problem of version controlling 300 gigs of files, I'll come back with an update whenever I decide on something and get it working (regardless of whether or not it's with Git Annex).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.