We use versions of Ruby, Collectd, Ant, Java (and more) that are not available in CentOS or EPEL repos. Up until now, our strategy for installing these has been kind of a hack:
write a (version-controlled) script for each package that downloads the source or binaries off the host site with curl onto the Puppet master and compiles the source code if necessary; repackages into gzipped tarball if necessary
use the Puppet file server to distribute the binaries to our servers, and Puppet manifests to unpack the tarball in /usr/local (or wherever)
Writing the scripts can be a pain, and they have to be updated if one of the sites we depend on for downloads changes their API. Also software is compiled separately in every environment, which seems wasteful and could potentially result in problems with missing compile-time dependencies (eg., Ruby: require 'readline' or require 'yaml' might work in some environments but not others)
So, I can think of two other options:
Just check in the custom-built third-party binaries into subversion and distribute them with the rest of our Puppet code. I'm worried it will severely impact the performance of checkouts and Puppet code pushes; we're looking at nearly 800MB (and growing) of third-party code. Plus it just feels wrong to check, like, multiple versions and architectures of Java side-by-side into version control.
Don't version control the binaries or write download scripts -- when we decide to upgrade Ruby, compile it on a dev box and upload the new package manually to all of our Puppet masters whenever we decide to upgrade. Except, what if the packages get wiped out? Or become out of sync on the different masters? Right now we can easily re-generate all of our custom packages from scratch.
Which of the three options is better? How do people typically manage custom-compiled / repackaged third-party software with Puppet? If you create a local Yum repository, do you version control the process you use to create the RPMs? What happens if your Yum repo is wiped out?