Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I would like to mirror the following Yum/RPM repositories at http://yum.puppetlabs.com/ :

The Puppet repository contains every Puppet product ever released and is quite large at about 8GB. I only need to mirror the newest versions of the files.

I have tried to mirror the repository using reposync --newest-only:

reposync --config=puppetlabs.repo.el6 --repoid=puppetlabs-products --repoid=puppetlabs-deps --newest-only --download_path=el/6 --quiet --downloadcomps

and this downloads the newest packages like I need. However, reposync doesn't automatically create the regular directory structure (x86_64, noarch, SRPMS, etc.) and doesn't mirror repodata.xml. As a result, my yum clients get errors like this:

[root@web1 ~]# yum --quiet install puppet
http://mirrors.example.org/pub/puppet/el/6/puppetlabs-deps/x86_64/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Trying other mirror.
Error: Cannot retrieve repository metadata (repomd.xml) for repository: puppetlabs-deps. Please verify its path and try again
[root@web1 ~]# 

Is there a way to programmatically mirror only the new files from a Yum repo and follow the standard repository directory structure?

share|improve this question
1  
I don't know offhand, but it's an interesting question. Personally I don't worry about it. 8GB is a tiny fraction of my 276GB of local mirrors... –  Michael Hampton Apr 23 at 4:18
    
Sure I know, why fret about 8GB. I'm just trying to be efficient :) In addition, sometimes I need to quickly set up another yum mirror of the EPEL or CentOS repos, and those are quite large. I really only need the latest N versions of the packages. –  Stefan Lasiewski Apr 23 at 17:48

2 Answers 2

reposync is the only reliable way to do this. You will need to create a small bash script and use reposync parameters (-a) to download each architecture in a separate folder and then run createrepo to generate the metadata.

Here is a small script that I have (it is running on Ubuntu but doesn't matter, you get the idea):

cat sync-repos

#!/bin/bash

reposync -n -c /etc/yum/yum.conf -p /repos/centos6 -d -r base -r updates -r extras -r centosplus -r contrib
createrepo -g /repos/centos6/base/repodata/comps.xml /repos/centos6/base
createrepo /repos/centos6/updates
createrepo /repos/centos6/extras
createrepo /repos/centos6/centosplus

reposync -n -c /etc/yum/yum.conf -p /repos -d -r vmware -r home_xtreemfs
createrepo /repos/vmware
createrepo /repos/home_xtreemfs

reposync -n -c /etc/yum/yum.conf -p /repos/vz -d -r openvz-utils -r openvz-kernel-rhel6
createrepo /repos/vz/openvz-utils
createrepo /repos/vz/openvz-kernel-rhel6

reposync -n -c /etc/yum/yum.conf -p /repos/nginx -d -r nginx-stable -r nginx-mainline
createrepo /repos/nginx/nginx-stable
createrepo /repos/nginx/nginx-mainline
share|improve this answer
    
Thanks for the tips. This sort of works, as reposync insists on appending the repoid to the directory structure. This means A command like reposync --config=puppetlabs.repo.el6 --repoid=puppetlabs-products --newest-only --arch=x86_64' creates a bizarrely named directory like 'x86_64/puppetlabs-products when I simply need 'x86_64/'. –  Stefan Lasiewski Apr 23 at 20:33
    
Yeah, I find that annoying as well but maybe if you make it a symlink or something like that it will work. On the other hand, if you have time to invest, reposync and createrepo are python scripts that you can modify to fit your needs :). –  Florin Asavoaie Apr 23 at 23:04

Another easy option to use for what you want to do, is spacewalk, is easy to use, you can establish a secure connection between the repo manager and your hosts, manage which packages you want to sync and provide to your hosts, schedule the syncs updates and patches and many more other cool features.

share|improve this answer
    
spacewalk looks nice, but it does way more then I need. We're already attempting provisioning using Puppet, Foreman and other systems. –  Stefan Lasiewski Apr 23 at 20:24

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.