7

Sorry if this is a newbie question. I try to describe the situation first, then the squid questin will come in.

The current Fedora/Centos installations have in their normal configuration files in /etc/yum.repos.conf a metalink which looks like this.

metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch

This metalink actually makes yum/dnf pick a "random" server site (picked by the server random geographically by world region according to the location by the client of the metalink).
This also is used in case of slow download to switch to the next better site.

I noticed due to docker builds a lot of downloads, that why i am considering a squid proxy which all machines must use. But this "random" strategy of yum/dnf, worries me. I do understand the intention of fedora/centos to distribute the load of of this free repositories, so actually I do not want to undermine this strategy

Can squid somehow intelligently detect, that the client just uses "another fedora/centos repo url" and intelligently cache this? The metalink list in itself seems to be pretty stable (it just changes the order when asked, but it the list itself seems to be the same).

Intention: Do not store 1000 copies of the same file, only because it is from a different server.

How would i do that with squid?

EDIT: Does somebody have experience using this http://wiki.squid-cache.org/Features/StoreID for caching of dnf/yum?

Improve this question
4
  • Have you considered running a local mirror, either private or public? If you do this, you can configure Fedora's MirrorManager service — the thing that generates those metalink results — so that requests from your network blocks get your server first.
    – mattdm
    Mar 9, 2017 at 15:41
  • local mirror is my fallback. But then i have to deal with "update the mirror frequently.". I'd rather leave this to the "official" sites.
    – Mandragor
    Mar 9, 2017 at 17:02
  • 1
    Yeah, I understand that. However, if you change your mind, it's pretty easy with Fedora Quick Mirror.
    – mattdm
    Mar 9, 2017 at 17:08
  • 1
    This really is a circumstance where you should run a local mirror. The more servers (or containers!) you have, the more it benefits you. I've had a private local mirror for quite a while, and MirrorManager just tells all the clients to hit it first.
    – Michael Hampton
    Mar 9, 2017 at 17:17

2 Answers 2

Reset to default
5

Answering my own question. Found out that squid has support for handling this kind of problem with the storeid_file_rewrite script. The only tricky thing is to get a valid list of urls, which represent the same repositories. Seems to work fine so far.

Added to squid.conf the following

store_id_program /usr/lib64/squid/storeid_file_rewrite /etc/squid/fedora.db
store_id_access allow localnet
store_id_access deny all

To get the content for the fedora.db (caching fedora 25 at this point in time) is some trickery with getting the urls from the mirrorlist

basearch="x86_64"
releasever=25
mirrorlist="https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearc
curl -s "$mirrorlist" >tmp.db

You need to convert the "url" in the "tmp.db" result into the format explained here http://wiki.squid-cache.org/Features/StoreID/DB. This can possibly automated (Any volunteers?)

Then you get something like this as "fedora.db", which is used in squid.conf above.

^http:\/\/ftp\.halifax\.rwth-aachen\.de\/fedora\/linux\/releases\/25\/Everything\/(x86_64\/[a-zA-Z0-9\-\_\.\/]+rpm)$    http://repo.mirrors.squid.internal/fedora/25/$1
^http:\/\/mirror2\.hs-esslingen\.de\/fedora\/linux\/releases\/25\/Everything\/(x86_64\/[a-zA-Z0-9\-\_\.\/]+rpm)$        http://repo.mirrors.squid.internal/fedora/25/$1
^http:\/\/fedora\.tu-chemnitz\.de\/pub\/linux\/fedora\/linux\/releases\/25\/Everything\/(x86_64\/[a-zA-Z0-9\-\_\.\/]+rpm)$      http://repo.mirrors.squid.internal/fedora/25/$1

... much more

EDIT: Alternative, a more dangerous path, but maybe also sufficient, a more global pattern matching like this:

\/fedora\/linux\/releases\/([0-9]+)\/Everything/x86_64\/(.*)$   http://repo.mirrors.squid.internal/fedora/releases/$1/$2
\/fedora\/linux\/updates\/([0-9]+)\/x86_64\/(.*)$       http://repo.mirrors.squid.internal/fedora/updates/$1/$2

Sources:

Improve this answer
3
  • With slightly more risk for false positives, you could try just using \/fedora\/linux\/releases\/[0-9]+\/Everything (and something to catch updates, too), rather than listing each mirror individually. Most mirrors should follow this same internal structure.
    – mattdm
    Mar 10, 2017 at 13:12
  • You are right. Updated
    – Mandragor
    Mar 10, 2017 at 13:43
  • An alternative take on the store_id_program is found in github.com/yevmel/squid-rpm-cache However none of this matters if the repos are using HTTPS! In that case you will need some sort of ssl_bumping. And that isn't so simple.
    – anthony
    Jul 16, 2022 at 2:57
0

You could consider using baseurl instead, as suggested here: http://serverascode.com/2014/03/29/squid-cache-yum.html

So use:

baseurl=http://mirror.fedoraproject.org/fedora/$releasever/os/$basearch/

instead of:

metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch

That avoids the issues with metalink by making things a bit less dynamic.

If you were going to use baseurl, I suspect another way of achieving what you are trying to do would be using apache or nginx (I would pick nginx) as a reverse proxy to the repo mirror.

Unless you were going to use squid for other purposes, I think I might in fact favour setting nginx up as a local mirror than use the internet mirrors as a backend/upstream.

Improve this answer
1
  • 1
    I cannot control all the user machines (and change their configs). But i can control the network, so that they must use the proxy. If I would have control over all configs i would use baseurl.
    – Mandragor
    Mar 9, 2017 at 15:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .