12 Jun 09 mrepo: RPM Repository Mirroring

Today I have finally found some time to catch up on my blogging. I’ve got a whole slew of backlogged information that I want to post, so hopefully my fingers (and my brain!) can keep up long enough to get it all published. The topic I wanted to start with today is mrepo: RPM Repository Mirroring made easy.

Mrepo is a tool developed by Dag Wieers of rpmforge fame. If you’ve ever run an RPM based distribution (Red Hat, Fedora, CentOS, etc) you’ve very likely heard of Dag Wieers. He keeps the package machine running and provides those packages not otherwise available in the core repositories. I read that he developed this tool to help streamline the creation and deployment of RPM mirrors and network install servers, which he so often needed in dealing with clients.

I used this exact setup to create and maintain an internal repository for our company which supplies network-based installation capabilities for Red Hat Enterprise 3, 4 and 5, CentOS 3, 4 and 5, Fedora 10 and 11, and openSUSE. This allows us to more quickly deploy new servers as well as saving bandwidth when applying errata to hundreds of servers and desktops.

Before I dive into the nuts and bolts of this operation there are a few things that are assumed with this setup. You should be aware of these so that you know what to expect and/or you can update your preferences accordingly.

1. The export and storage location I prefer is /var/ftp/pub, with the DVD images stored in a subdirectory of /var/ftp/pub/.iso/. This allows client access via FTP without any FTP server customizations, but also HTTP by way of symbolic links and NFS with basic configuration.
2. The export location also takes into consideration the fact that many people put /var/ on its own partition. A mirror will require decent amount of space, and this allows it to be separate from the rest of the file system.
3. If you want your mirror available via FTP, HTTP and NFS you’ll need to install the vsftpd and httpd packages.
4. Steps 3 and 4 are only required if you plan on mirroring Red Hat Network errarta. This requires a valid RHN account and available entitlement!
5. The mrepo wiki page on my personal wiki is available to the public for additional reference and discussion.

With those details out of the way let’s get started!

Step 1: Build and Install The Package

The latest codebase for mrepo is not available in package format (at least that I was able to find), so we’ll have to build the package from svn. I have done this a few times and it is very painless. Simply follow the below commands and you’ll end up with an installed mrepo RPM.

mkdir -p ~/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
svn co http://svn.rpmforge.net/svn/trunk/tools/mrepo/ ~/mrepo-0.8.6svn/
tar cjvf ~/rpmbuild/SOURCES/mrepo-0.8.6svn.tar.bz2 ~/mrepo-0.8.6svn/
echo "%_topdir ~/rpmbuild/" > ~/.rpmmacros
cd ~/mrepo-0.8.6svn/
rpmbuild -ba mrepo.spec
yum localinstall ~/rpmbuild/RPMS/noarch/mrepo-0.8.6svn*.noarch.rpm

If everything goes according to plan you should end up with an mrepo-0.8.6svn*.noarch.rpm in your ~/rpmbuild/RPMS directory, and it should also be installed. If you don’t have these, or you ran into other problems leave a comment and we’ll see what we can figure out.

Step 2: Create The Public Directory Structure

I’m guessing the reason that you want to create and export an RPM repository is that you want to share packages with others, or perhaps you want the ability to do network-based installations. This means you’ll want to make this content public. You can choose a different location for your export if you’d like, but I prefer to put my content in /var/ftp/pub/, so here are the directories I’d create if I wanted to make available a CentOS, Red Hat Enterprise and Fedora repository, including the latest version and both major architectures.

mkdir -p /var/ftp/pub/.iso/{rhel,centos}5-{i386,x86_64}
mkdir -p /var/ftp/pub/.iso/fedora11-{i386,x86_64}

Step 3: Generate System ID For Use With RHN (Optional)

If you’re going to be exporting Red Hat Enterprise images and packages you’re going to need to generate a System ID for each release and architecture you want to gather updates for. This will allow your machine (even a non-Red Hat Enterprise machine) to connect to and gather updates from the Red Hat Network. Note: This does require a valid RHN username, password and entitlement!

gensystemid -u RHN_USERNAME -p RHN_PASSWORD --release=5Server --arch=i386 /var/ftp/pub/.iso/rhel5-i386
gensystemid -u RHN_USERNAME -p RHN_PASSWORD --release=5Server --arch=x86_64 /var/ftp/pub/.iso/rhel5-x86_64


Note: Be sure to change the paths above to match your own if you did not follow step two exactly.

Step 4: Minor Tweaks Needed if Run On CentOS (Optional)

I have found there are a few minor things that need to happen in order for a CentOS based machine to mirror updates from the Red Hat Network. They are fairly simple, but are required in order for the CentOS machine to properly and securely connect to the Red Hat Network. Note: If you are not planning on mirroring Red Hat Enterprise packages you can skip this step.

echo "up2date default" >> /etc/sysconfig/rhn/sources
echo "uuid[comment]=Universally Unique Identifier for this server" >> /etc/sysconfig/rhn/up2date-uuid
echo "rhnuuid=$(uuidgen)" >> /etc/sysconfig/rhn/up2date-uuid
mkdir /usr/share/rhn
cp RHN-CA-CERT /usr/share/rhn/

Note: The last command will not work out-of-the-box because the RHN-CA-CERT file is not found by default on a CentOS machine. You will need to copy this from a Red Hat Enterprise machine (located in /usr/share/rhn/RHN-CA-CERT) and place it on your CentOS machine.

Step 5: Download The .iso Images

We’re now ready to download and install images we’re going to want to export and use. Below are a few example commands for CentOS and Fedora. After these are downloaded be sure to place them in their appropriate directory, located in /var/ftp/pub/.iso/. The full path should match that found in step 6 below for “iso=”

CentOS 5.3:

wget -c http://mirrors.kernel.org/centos/5.3/isos/i386/CentOS-5.3-i386-bin-DVD.iso
wget -c http://mirrors.kernel.org/centos/5.3/isos/x86_64/CentOS-5.3-x86_64-bin-DVD.iso


Fedora 11:

wget -c http://mirrors.kernel.org/fedora/releases/11/Fedora/i386/iso/Fedora-11-i386-DVD.iso
wget -c http://mirrors.kernel.org/fedora/releases/11/Fedora/x86_64/iso/Fedora-11-x86_64-DVD.iso

Your milage may vary with those mirrors, but the basic idea is to download (or if you already have downloaded, copy) the DVD images and place them into your previously created export directory of /var/ftp/pub/.iso/{$distro$releasever-$basearch}.

Step 6: Create The Config File

We’re almost done! I hope you’ve stayed with me so far. The last thing we need to do before we can start downloading our updates is create the configuration file. This file will detail three main things:

1. Where are the DVD images we want to export?
2. Where do we want to export them?
3. What mirror(s) do we want to pull our updates/errata from?

Once we’ve defined these items we’ll be ready to kick off mrepo and populate our mirror.

The mrepo package that we built earlier should come with a default config file. To make things simple we’ll just customize the existing file and then we’ll be ready to go. The config file can be found at /etc/mrepo.conf, and here is an example config with entries for CentOS, Fedora and Red Hat Enterprise:

[main]
srcdir = /var/ftp/pub/.iso
wwwdir = /var/ftp/pub
confdir = /etc/mrepo.conf.d
arch = i386 x86_64
mailto = root@localhost
smtp-server = localhost
rhnlogin = username:password

[centos5-i386]
name = CentOS $release ($arch)
release = 5.3
arch = i386
iso = /var/ftp/pub/.iso/centos5-i386/CentOS-$release-$arch-bin-DVD.iso
updates = rsync://mirrors.kernel.org/centos/$release/updates/i386/

## centos 5 x86_64
[centos5-x86_64]
name = CentOS $release ($arch)
release = 5.3
arch = x86_64
iso = /var/ftp/pub/.iso/centos5-x86_64/CentOS-$release-$arch-bin-DVD.iso
updates = rsync://mirrors.kernel.org/centos/$release/updates/x86_64/

## rhel 5 i386
[rhel5-i386]
name = Red Hat $release Server ($arch)
release = 5.3
arch = i386
iso = /var/ftp/pub/.iso/rhel5-i386/rhel-$release-server-$arch-dvd.iso
updates = rhns:///rhel-$arch-server-5

## rhel 5 x86_64
[rhel5-x86_64]
name = Red Hat $release Server ($arch)
release = 5.3
arch = x86_64
iso = /var/ftp/pub/.iso/rhel5-x86_64/rhel-$release-server-$arch-dvd.iso
updates = rhns:///rhel-$arch-server-5

## fedora 11 i386
[fedora11-i386]
name = Fedora $release ($arch)
release = 11
arch = i386
iso = /var/ftp/pub/.iso/fedora11-i386/Fedora-$release-$arch-DVD.iso
updates = rsync://mirrors.kernel.org/fedora/updates/$release/$arch/

## fedora 11 x86_64
[fedora11-x86_64]
name = Fedora $release ($arch)
release = 11
arch = x86_64
iso = /var/ftp/pub/.iso/fedora11-x86_64/Fedora-$release-$arch-DVD.iso
updates = http://mirrors.kernel.org/fedora/updates/$release/$arch/

Once you have made the needed changes to your configuration you’re ready to kick off the mirror and export process. Based on what we have in the configuration the next step will do the following:

• Loop-Mount and export the DVD images to /var/ftp/pub/. It will create subdirectories using the bracketed title we used in the config file (ie; /var/ftp/pub/fedora11-x86_64/)
• Mirror the public updates contents locally from the locations defined by “updates =”
• Create two subdirectories within our wwwdir path for RPMS.os and RPMS.updates.

Step 7: Create The Repositories

If you’ve made it through each of the previous steps without hitting any roadblocks you’re now ready to initiate your mirror creation.

mrepo -guvvv

Note: The command above suggests very verbose output (vvv) for the first run. This way you can see what it is doing and the progress it is making. For future runs you can omit the verbose options.

Tags: CentOS, mirror, Red Hat, repository, RHN, RPM