8

I would like to mirror the following Yum/RPM repositories at http://yum.puppetlabs.com/ :

The Puppet repository contains every Puppet product ever released and is quite large at about 8GB. I only need to mirror the newest versions of the files.

I have tried to mirror the repository using reposync --newest-only:

reposync --config=puppetlabs.repo.el6 --repoid=puppetlabs-products --repoid=puppetlabs-deps --newest-only --download_path=el/6 --quiet --downloadcomps

and this downloads the newest packages like I need. However, reposync doesn't automatically create the regular directory structure (x86_64, noarch, SRPMS, etc.) and doesn't mirror repodata.xml. As a result, my yum clients get errors like this:

[root@web1 ~]# yum --quiet install puppet
http://mirrors.example.org/pub/puppet/el/6/puppetlabs-deps/x86_64/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Trying other mirror.
Error: Cannot retrieve repository metadata (repomd.xml) for repository: puppetlabs-deps. Please verify its path and try again
[root@web1 ~]# 

Is there a way to programmatically mirror only the new files from a Yum repo and follow the standard repository directory structure?

2
  • 2
    I don't know offhand, but it's an interesting question. Personally I don't worry about it. 8GB is a tiny fraction of my 276GB of local mirrors... Apr 23, 2014 at 4:18
  • Sure I know, why fret about 8GB. I'm just trying to be efficient :) In addition, sometimes I need to quickly set up another yum mirror of the EPEL or CentOS repos, and those are quite large. I really only need the latest N versions of the packages. Apr 23, 2014 at 17:48

4 Answers 4

8

reposync is the only reliable way to do this. You will need to create a small bash script and use reposync parameters (-a) to download each architecture in a separate folder and then run createrepo to generate the metadata.

Here is a small script that I have (it is running on Ubuntu but doesn't matter, you get the idea):

cat sync-repos

#!/bin/bash

reposync -n -c /etc/yum/yum.conf -p /repos/centos6 -d -r base -r updates -r extras -r centosplus -r contrib
createrepo -g /repos/centos6/base/repodata/comps.xml /repos/centos6/base
createrepo /repos/centos6/updates
createrepo /repos/centos6/extras
createrepo /repos/centos6/centosplus

reposync -n -c /etc/yum/yum.conf -p /repos -d -r vmware -r home_xtreemfs
createrepo /repos/vmware
createrepo /repos/home_xtreemfs

reposync -n -c /etc/yum/yum.conf -p /repos/vz -d -r openvz-utils -r openvz-kernel-rhel6
createrepo /repos/vz/openvz-utils
createrepo /repos/vz/openvz-kernel-rhel6

reposync -n -c /etc/yum/yum.conf -p /repos/nginx -d -r nginx-stable -r nginx-mainline
createrepo /repos/nginx/nginx-stable
createrepo /repos/nginx/nginx-mainline
3
  • Thanks for the tips. This sort of works, as reposync insists on appending the repoid to the directory structure. This means A command like reposync --config=puppetlabs.repo.el6 --repoid=puppetlabs-products --newest-only --arch=x86_64' creates a bizarrely named directory like 'x86_64/puppetlabs-products when I simply need 'x86_64/'. Apr 23, 2014 at 20:33
  • Yeah, I find that annoying as well but maybe if you make it a symlink or something like that it will work. On the other hand, if you have time to invest, reposync and createrepo are python scripts that you can modify to fit your needs :). Apr 23, 2014 at 23:04
  • 5
    probably wasn't available at the time of the post, but for the benefit of future readers: --norepopath Don't add the reponame to the download path. Can only be used when syncing a single repository (default is to add the reponame). May 11, 2017 at 22:25
4

You can do this with pulp and the yum rpm distributor plugin.

When congifguring a new repo, to get only one verison of each rpm, set the retain_old_count retain_old_count parameter

retain_old_count
Count indicating how many old rpm versions to retain; by default it will 
download all versions available.

So something along the line of:

$ pulp-admin rpm repo create \
          --repo-id=rhel6-puppet-products \
          --relative-url=rhel6-puppet-products \
          --feed=http://yum.puppetlabs.com/el/6/products/ \
          --retain-old-count 1
$ pulp-admin rpm repo sync run  \
          --repo-id=rhel6-puppet-products \

Should achieve what you want. There is a quick start guide which should give you an idea of how the thing works, in case you have not tried it before.

1
  • This answer needs more context. Why, for instance, should a user download, install, provision and fight with pulp for the rest of their sanity instead of using a fixed procedure with reposync? Ideally, you may want to show what the user gets as a benefit once paying the cost of tearing every bit of their hair out when pulp ruins at least one day every month. Oct 2, 2023 at 7:54
0

Another easy option to use for what you want to do, is spacewalk, is easy to use, you can establish a secure connection between the repo manager and your hosts, manage which packages you want to sync and provide to your hosts, schedule the syncs updates and patches and many more other cool features.

1
  • spacewalk looks nice, but it does way more then I need. We're already attempting provisioning using Puppet, Foreman and other systems. Apr 23, 2014 at 20:24
0

Edit /etc/sysconfig/uln-yum-mirror Change ALL_PKGS = 0

0 -> download latest only, 1 -> download all versions

1
  • 6
    Your answer really needs more context to be useful. Are you able to edit it? For example, how does this provide a repository? Is this answer in addition to another answer?
    – Tim
    Jun 18, 2017 at 23:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .