0

To avoid the XY problem, let me first describe the situation.

We have a client project of unique circumstances. We have a relatively modern software stack (Keras DNN stuff) that needs to run on a client's system. This system, a Cloudera CentOS 6 cluster, in production, is airgapped for security. We can't guarantee this thing has ever seen the internet, ever.

We developed a bash script which installs the requisite packages from disk using RPM and tested on our local simulated (containerized) cluster (YUM fails because the repo database is not up to date). After some fiddling, we were able to get Keras running without so much as a packet from the internet.

The client has his own virtualized system set up which is supposed to be pretty close to the actual cluster (in terms of config). When he ran it, however, total catastrophe. Lots of errors like:

(for installing glibc-common-2.12 and friends via sudo rpm -Uvh glibc-common-2.12-1.209.el6_9.1.x86_64.rpm glibc-2.12-1.209.el6_9.1.x86_64.rpm glibc-headers-2.12-1.209.el6_9.1.x86_64.rpm glibc-devel-2.12-1.209.el6_9.1.x86_64.rpm )

warning: ./rpm/glibc-common-2.12-1.209.el6_9.1.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
error: Failed dependencies
    tzdata >= 2015g-4 is needed by glibc-common-2.12-1.209.el6_9.1.x86_64.rpm
    libc.so.6(GLIBC_2.13)(64bit) is needed by (installed) util-linux-2.23.2-26.el7.x86_64
    libc.so.6(GLIBC_2.13)(64bit) is needed by (installed) systemd-219-19.el7.x86_64
    ...(more of the same)...
    libc.so.6(GLIBC_2.13)(64bit) is needed by (installed) xz-libs-5.1.2-12alpha.el17.x86_64

(Or running the command: sudo rpm -Uvh gcc-c++-4.4.7-18.el6.x86_64.rpm gcc-4.4.7-18.el6.x86_64.rpm libstdc++-4.4.7-18.el6.x86_64.rpm libstdc++-devel-4.4.7-18.el6.x86_64.rpm )

libstdc++.so.6(GLIBCXX_3.4.15)(64bit) is needed by (installed) {name of a package}
libstdc++.so.6(GLIBCXX_3.4.15)(64bit) is needed by (installed) {name of another package}
...

X is needed by Y is the most common error, but we also see ones like

openssl < 1:1.0.1-0.3.beta3 is obsoleted by (installed) openssl-libs-1:1.0.1e-42.el4.9x86_64

and

file /etc/rpm/macros.ghc-srpm from install of epel-release-6-8.noarch conflicts with file from package redhad-rpm-config-9.1.0-68.el7.centos.noarch

In particular, a lot of conflicts are stemming from

  • libc.so.6
  • libm.so.6
  • libgmp.so.3
  • libmpfr.so.4
  • libstdc++.so.6
  • libffi.so.6

which are all super crucial libraries, and I 1) can't assume any particular version will be on the cluster, and 2) can't be monkeying around with, lest one of them breaks.

I'm more of a machine learning CS and my linux skills are enough to set up and service CUDA boxes and such, but this is starting to be deep water for me, so any input, even simple stuff, is appreciated. Is there any way to create a separate library environment where we can install the necessary deps without disturbing the ones already there? I know chroot is a thing but I have no idea how to properly wield it.

tl;dr - dependency hell on an airgapped system with no remote admin ability and a crusty old, poorly-maintained build.

Many thanks!

1

I have to work in airgapped environments very often.

The best approach I have found so far is to have a virtual machine with EXACTLY the same environment (i.e. exactly the same version for all packages, if possible with its installation following the same process used for the production environment) - give that VM internet access, and use the appropriate package manager to DOWNLOAD ONLY all required packages.

To download only packages and not install them, I use:

yum install --downloadonly <requiredpackage>

This works out of the box at least with CentOS 6 - in other RPM based distribution you may need to install the yum-downloadonly package, as this functionality is provided by a yum plug-in.

Then, I archive all the downloaded packages in a .tar.gz file, transfer that archive to the production system, decompress in a temporary directory and install packages using:

rpm -i <directory>/*.rpm

It is also important to use point releases instead of only the major CentOS version or the "$releaseversion" variable in the /etc/yum.repos.d/*.repo configuration files, as the repositories with major version get updates over time. This point release shall match the installed version on the VM used for downloading and on the airgapped system.

For example, use:

http://mirror.centos.org/centos/7.7.1908/os/x86_64/

instead of

http://mirror.centos.org/centos/7/os/x86_64/

as the latter will have packages updates until CentOS 7 reaches its end of maintenance (November 30, 2020 - according to the CentOS FAQ) .

From the description you made, it seems that the virtual machine you mention has been somehow updated from the CentOS repositories, so you're having some version dependency conflicts.

Hope this helps.

7
  • downloadonly and rpm -i is exactly how the script runs currently. > From the description you made, it seems that the virtual machine you mention has been somehow updated from the CentOS repositories, so you're having some version dependency conflicts. I'm glad you mentioned this, as I suspected exactly this, but not enough confidence or know-how to do anything about it. But I'll investigate this deeper. – DeusXMachina Apr 24 '17 at 15:46
  • If you can, and have some time to spend on this, try to install a new fresh VM following the original process used to install the production server, and test the installation wihout internet access on that machine. – niglesias Apr 24 '17 at 15:55
  • Also, note you're trying to install -Uvh glibc-common-2.12-1.209.el6_9.1.x86_64.rpm and the RPM system says libc.so.6(GLIBC_2.13)(64bit) is needed by (installed) util-linux-2.23.2-26.el7.x86_64 – niglesias Apr 24 '17 at 15:57
  • The blank Cloudera VM downloads GLIBC_2.12 when I call that dependency (with zero yum updates). I highly suspect now that the Client's laptop is running something updated. We have a yum debug dump now though so we should be able to back-track and figure out the exact production build. – DeusXMachina Apr 24 '17 at 16:01
  • Run rpm -qav on both environments and check for differences – niglesias Apr 24 '17 at 16:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.