Cross Linux From Scratch

This course will become read-only in the near future. Tell us at community.p2pu.org if that is a problem.

RPM [July 1, 2012, 9:16 p.m.]


Since this package has a lot of little dependencies, I decided to throw in a quick table of contents. If you know you have everything, just go ahead and jump right on down to the RPM section.

Windows Users: Cygwin has successfully built the rpm package. You'll want to install rpm, rpm-build, and rpm-doc. I tried unsuccessfully to port rpm into windows and build it natively; you can read more about it in the "Windows Environment" task in the "gnulib" section, along with some generally helpful porting hints.

So windows users, you'll want to go through this entire task. Building the beecrypt package introduces important concepts, and the rpm section has tips for running Cygwin's RPM with MinGW.

Why Package? Aren't we using Git?

Even if you don't want to use RPM to package with, you might want to read through this task. The strategies I'll be going over can be used with any source-based package management. 

In addition to version control, you'll also want some sort of package management. Packaging offers a few advantages simple version control doesn't:

  • It's very easy to build a package once, and then install it on other computers, without having to rebuild
     
  • If you are compiling a long line of packages, and you mess up on package 3 which 6 and 7 depend on but 4 and 5 don't, you can uninstall 3, 6, and 7 without affecting 4 and 5. This would be very difficult using version control alone.
     
  • It's very easy to make minor changes to packages if you have to reinstall them later, such as compiling in a new feature
     
  • It's very easy to upgrade packages to newer versions without destabilizing your whole OS
     
  • Finally, just about any package management system you can use (including simple shell scripts; check out Sorcery and Sourcemage some time, or my personal favorite source-based package repository, slackbuilds.org) probably has a community with pre-written source packages and even pre-compiled binaries. Often, installing software on Linux can be just as easy as installing software on windows, and as an added bonus you have more control and don't feel dirty signing any EULA contracts. Packaging means Linux can be "double-click and run" too.

 

Packaging and version control complement each other well. Version control makes it very easy to keep track of configurations and metadata. If you changed something in your spec (build instructions) to work with a specific version of a package, for example, or you want to use an old version of a startup script, version control is a very good tool for this purpose. And if you don't want to have to repeat a lot of steps creating software, packaging is a good tool to use here. With both combined, you'll almost never have to worry about things being deleted, and your system will almost always be recoverable. And best of all, you won't need hundreds of Gigabytes to back up your OS; your OS will constantly be backed up as you use it.

If you haven't picked up on it by now, I'm also a huge fan of a third kind of backing up: Read-only media. Git and RPM work great for solving just about any software pickle you can find yourself in. But if your hard disk itself ever gets damaged, or you accidentally do something catastrophic to your software, then they'll fall short. The reason why CDs and DVDs are great, is that you write to them once, and then you can't overwrite them. And it's really easy to port stuff over to other computers without setting up a network or using Dropbox.

Another great solution is "cloud storage". If you send your data over a network to another computer (including the servers housing your e-mail or Dropbox stuff), that's what's called "write restricted". It means that there are enough steps in place that you really have to make an effort to permanently delete something. With e-mail and Dropbox, this also serves as a kind of version control as well. With Linux, it's very easy to have total control over every single computer your data touches, so you don't even have the security and privacy concerns you'd have using e-mail.

Since the CLFS and CBLFS books are, in a sense, a source-based package management system in and of themselves, I'd highly recommend going that route. The difference between source-based package management and binary-based package management, is that with source-based packaging, you download and build the package from source fresh every time you want to install it, and with binary-based packaging, you just download a package you already built and install that instead. Source-based is more flexible and you don't get bogged down with dependencies. Binary-based is faster and easier. Windows setup.exes are an example of binary-based package management.

You are free, of course, to use a binary-based package management system if you want. That's going to be a little more in depth than anything I'll cover here, but the gist of it is that you'll want to write your own software that uses a database of some kind (probably SQLite) to keep track of compatibilities. Git is great for distributing your packages online once you've done that and keeping track of dependency sets. I'm a lighttpd fan, but the good old Apache httpd server has a lot of built-in support for git. I won't be showing you how to modify the database that RPM itself uses to track binary packages and dependencies because, frankly, I don't know how and don't know where to find this information online. I've tried tracking it down in the past unsuccessfully. So I'll be installing everything with the --nodeps flag as a result.

Alrighty, I'm done yappin. Let's get to work.

Beecrypt

RPM has one dependency, Beecrypt. Beecrypt and RPM have CBLFS pages here:

http://cblfs.cross-lfs.org/index.php/Beecrypt

http://cblfs.cross-lfs.org/index.php/RPM5

One other dependency you'll want for beecrypt is JDK. Make sure JAVA_HOME is set, and $JAVA_HOME/bin is on your path. Here's a guide on doing that:

http://www3.ntu.edu.sg/home/ehchua/programming/howto/Environment_Variables.html

To get this to build on MinGW/MSYS, I had to patch a file. I found out how to patch the file by going to the project's homepage, which I found on the project's CBLFS page.The patch was designed not to interfere with your build process in any environment, but was designed with MinGW in mind. The patch can downloaded from here, Since I'll be going over it later, I'll also paste it right into these instructions.

file beecrypt-4.2.1-mingw_make_dlls.patch :

--- beecrypt-4.2.1/md4.c.orig    2012-06-01 18:33:53 -0500
+++ beecrypt-4.2.1/md4.c         2012-06-01 18:35:47 -0500
@@ -1,4 +1,18 @@
 /*
+http://sourceforge.net/projects/beecrypt/forums/forum/27737/topic/3351667
+
+Developer suggested bug fix for the following problem:
+
+.libs/libbeecrypt.dll.a.libs/md4.o:md4.c:(.text+0x571):
+ undefined reference to \`_imp__mpsetw'
+
+Enable by setting -DMINGW_MAKEDLLS in CFLAGS
+*/
+#ifdef MINGW_MAKEDLLS
+#define BEECRYPT_DLL_EXPORT
+#endif /* MINGW_MAKEDLLS */
+
+/*
  *
  */

 

To create the patch, I followed this four step process

  1. Copy the file you're changing to the filename plus .orig

    cp md4.c{,.orig}
     
  2. Edit the file you're changing

    vim md4.c
     
  3. Change to your build directory

    cd ~/clfs/build
     
  4. Run the diff tool on the original and changed files  to create the patch

    diff -du beecrypt-4.2.1/md4.c{.orig,} > \
      ~/clfs/patches/beecrypt-4.2.1-mingw_make_dlls.patch

 

Note that the comma was before .orig the first time and after .orig the second time. Also note the naming convention I used for the patch; it's packagename-version-description.patch.

To learn more about patches, go ahead and open the patch in a text editor. It contains a URL. Go ahead and open that URL too. You'll notice I was patching a c file. You don't have to be a full-fledged c programmer to benefit from this information, but I wanted to go ahead and walk you through the changes I made.

First, every time you see a line starting with

+

(your browser might be wrapping long lines, but most of them start with a +) that means that it's something that is in the changed file and not the original - in other words, something that was added. If something was removed, that would mean starting with a

-

Go ahead and ignore the + or - starting the lines above now that you know what they mean.

Second, everything between
/*

and

*/

is commented out. Just like using # in bash, the c compiler ignores everything between /* and */. I added some helpful information in-between those tags in case someone got hold of my patch and wanted to learn more about it.

Third,

#ifndef MINGW_MAKEDLLS

and

#endif

means in plain english, "only do this stuff if we're using MinGW". It pulls this off by first checking to see if something called MINGW_MAKEDLLS exists. Later I'll show you how I set this, but it's enough to know right now that it only exists if we're really using MinGW. I put this in there so that the patch wouldn't interfere with the linux/mac/bsd people.

Finally,

#define BEECRYPT_DLL_EXPORT

is just word-for-word what they told me to add in the online help.

Our shell script for building beecrypt, which you can download here, is as you'll notice a little more complicated:

file beecrypt.sh:

#!/bin/bash

# Exit if anything goes wrong immediately
set -e

# These should stay the same between packages
BUILDDIR="/home/sean/clfs/build"
PATCHDIR="/home/sean/clfs/patches"
SOURCEDIR="/home/sean/clfs/sources"
PREFIX="/usr"
SYSCONFDIR="/etc"
CUID="${UID}"
if [ -f /msys.bat ]
then
  PREFIX="/mingw"
  net localgroup Administrators | grep "$(whoami)" && CUID="0"
fi

# These change with every package
PACKNAME="beecrypt"
VERSION="4.2.1"
TARFMT="tar.gz"
TARBALL="${PACKNAME}-${VERSION}.${TARFMT}"
PACKDIR="${PACKNAME}-${VERSION}"
SOURCE="http://downloads.sourceforge.net/${PACKNAME}/${TARBALL}"
MD5SUM="8441c014170823f2dff97e33df55af1e"

# Specify options here
export CFLAGS="-I${JAVA_HOME}/include"
export CPPFLAGS="${CFLAGS}"

# Set compatibility options here
if [ -f /msys.bat ]
then
  export CFLAGS="${CFLAGS} -I${JAVA_HOME}/include/win32 -DMINGW_MAKEDLLS"
  export CPPFLAGS="${CFLAGS}"
  export LDFLAGS="-lgomp"
fi

# Description
cat << "EIO"
"Introduction to Beecrypt

BeeCrypt is an ongoing project to provide a strong
 and fast cryptography toolkit. Includes entropy sources,
 random generators, block ciphers, hash functions,
 message authentication codes, multiprecision integer
 routines, and public key primitives.

Project Homepage: http://beecrypt.sourceforge.net/
Optional Depdencies:
  GCC (for GCJ Java Compiler) or JDK
  Python

Contents:
  Installed Directories: /usr/include/beecrypt
  Installed Libraries:   libbeecrypt.{a,so}

Short Descriptions
  libbeecrypt.{a,so}: a library that contains functions
                      for strong and fast cryptography."
EIO

if [ "${CUID}" != "0" ]
then
  # Download the source package
  cd $SOURCEDIR
  [ ! -f $SOURCEDIR/$TARBALL ] && wget $SOURCE
  [ "$(md5sum ${TARBALL} | awk '{print $1;}')" != "${MD5SUM}" ] \
    && echo "Error: this doesn't look like the right package" \
    && exit 1

  # Extract the source
  cd $BUILDDIR
  rm -rf $PACKDIR # We always want to insist on a clean build
  tar xvf $SOURCEDIR/$TARBALL
  cd $PACKDIR

  # Patch
  patch -Np1 -i "${PATCHDIR}/${PACKDIR}-mingw_make_dlls.patch"

  # Configure and compile the package:
  cat > used_config.txt << EIO
--prefix=$PREFIX
--sysconfdir=$SYSCONFDIR
EIO
  ./configure "`cat used_config.txt`"
  make

  echo "Successfully built, run this script as root to install"

# Install the package:
elif [ ! -d "${BUILDDIR}/${PACKDIR}" ]
then
  echo "Run this as an ordinary user first"
  exit 1
else
  cd "${BUILDDIR}/${PACKDIR}"

  # Run any tests

  # Install the package
  make install

  # Install documentation

fi

Note: I made some cosmetic changes to the above script to aid the wiki's built-in syntax highlighter. Both the above script and the one you download do exactly the same thing.

This is done so that we can copy and paste this whole script to create new packages while only changing a few lines. I'll go ahead and walk you through the sections really quickly.

# These should stay the same between packages

Everything in this section you can edit one time, to change "sean" to your username, and then never touch again no matter how many times you copy it. It's cross platform (although let me know if there's a better way to check to see if you're inside the MinGW/MSYS environment in the comments section below).

# These change with every package

This is where we take advantage of some common conventions. Usually with a unix source package, you'll download something that looks like "beecrypt-4.2.1.tar.gz".

  • The first part, "beecrypt", is the package name, which we'll call PACKNAME
  • The second part, "4.2.1", is the version, which we'll call VERSION
  • The third part, "tar.gz", is the compression format, and might also be "tar.bz2", "tar.xz", or rarely, "tgz" or "zip". We'll call this TARFMT.

 

Usually this extracts to a folder called "PACKNAME-VERSION" (so in my case, "beecrypt-4.2.1"). We set these here not just to save us keystrokes later, but to avoid typos. If you get it right here, you get it right everywhere you use PACKNAME. It's also right at the beginning, so it's easy to change to whatever package we're installing.

We also have something called "MD5SUM" here. This is what we call an "integrity check", which is just a way to check to make sure the source you downloaded is really the right source. I got this by first manually downloading the source I wanted to use, and then running this on it:

md5sum $PACKNAME-$VERSION.$TARFMT | awk '{print $1;}'

This comes in really handy to prevent, for instance, you accidentally grabbing the wrong version of a file.

# Specify options here

For most packages, you can leave these lines blank. If you need to set CFLAGS, CXXFLAGS, LDFLAGS, or any other environment variables, you can set them here so they're close to the top of the file. You'll see this happen from time to time in the CLFS book, especially early on. In this case, we need them to include java headers.

# Set compatibility options here

Another one that can usually be left blank. I set stuff here so that the script would be run a different way if you were running MinGW. Incidentally, remember earlier how I said I'd show you how I defined MINGW_MAKE_DLLS only if we're using MinGW? This is where I did that, and now you know :)

# Description
cat << EIO
...
EIO

I copied and pasted the description from the CBLFS page right in here in place of the ... . This is completely optional. It's nice for me because I don't have to go to the CBLFS page if I want to know what beecrypt does later.


if [ "${CUID}" != "0" ]
then
  ...

This causes the script to run a different way if you're root, than it does if you're an ordinary user. It's useful if, like we'll be doing here, you're building packages as an ordinary user and installing them as root.

  # Download the source package

Exactly "what it says on the tin". Note that we only download the source package if we don't already have it. Also, we'll go ahead and run that integrity check here. As you can see, we use the same command to run the integrity check as we did to get MD5SUM to begin with. So if you forget, it's right here. If anything goes wrong, we bail out right here.

  # Extract the source
  ...
  # Patch
  ...
  # Configure and compile the package
  ...
 

We copy the instructions from the CBLFS page in here, with a few tweaks.

First, you'll notice I have everything organized into three sections. It's okay if you don't do this. I did this to make the transition to RPM specs a little more smooth later on.

Second, I adding the patch. One thing you might have to tweak here from time to time, is changing -Np1 to -Np0 or some other number. For us here we won't need to change it, but not everyone follows the convention of patching from an outer directory like we did.

Last, I also dumped all the configure options to a text file. This is an extremely useful practice if a package fails to build and you're experimenting with configure flags, and I'd highly encourage you to do this. The cat tool combined with the backticks will compress everything onto a single line for you automatically, so you don't have to worry about ending lines with \.

Again, if you want to just get something working, it's okay to copy instructions from the CLFS or CBLFS book in this section verbatim and ignore all the formatting and categorizing. It's up to you.

  echo "Successfully built, run this script as root to install

# Install the package:
elif [ ! -d $BUILDDIR/$PACKDIR ]
then
  echo "Run this as an ordinary user first"
  exit 1
else
  cd $BUILDDIR/$PACKDIR

  # Run any tests

  # Install the package
  make install

  # Install documentation

fi

First, we let the ordinary user know that everything worked out okay and we're ready to install. Next, we throw in a quick check to make sure we didn't try running this script as root before running it as an ordinary user. Finally, we copy whatever installation instructions there are in the CLFS or CBLFS book page (usually just "make install") in. I left a spot in there for you to run things like "make check" if you want to. Sometimes there are also special installation instructions for installing documentation. Here there aren't any.

That's it for beecrypt. If anything went wrong for you, please refer to the task for building git.

Popt

Even though this is listed in the CBLFS page as being "recommended", some systems will require you to have popt before RPM will install.

For unix-likes, Popt's CBLFS page is here:

http://cblfs.cross-lfs.org/index.php/Popt 

Chances are you already have it; look for /usr/include/popt.h. If it's there, you have popt. For windows, you'll want to grab popt-1.8-1-lib from the GnuWin32 site, here:

http://sourceforge.net/projects/gnuwin32/files/popt/1.8-1/popt-1.8-1-lib.zip/download

The next step on windows is to, in an admin shell, do this (replace sean with your username)

cd /mingw
unzip /home/sean/clfs/sources/popt-1.8-1-lib.zip

Either way, since the case where we'd build it ourselves is rare, we won't be making a popt.sh script. If you are on a rare system that doesn't have popt and you make a popt.sh script, feel free to share it in the comments section below. Otherwise, we're ready to build rpm.

RPM

Windows users, please see the note at the top, then keep reading.

Since RPM is going to be Cygwin dependent, and we've done everything else using MinGW, we're going to set up Cygwin as our development environment instead of MSYS. We'll do that with the following commands in an admin Cygwin shell (and assuming MinGW/MSYS is installed to C:/MinGW:

cd /
ln -s /cygdrive/c/MinGW mingw
cd /usr/share
ln -s /cygdrive/c/Program\ Files ProgramFiles
ln -s /cygdrive/c/Program\ Files/Microsoft\ Visual\ Studio\ 10.0/VC msvc
cd /etc
sed -e 's@^PATH="@&/mingw/bin:@" -i profile
cat >> profile << "EIO"
export CFLAGS="${CFLAGS} -I/mingw/include"
export CPPFLAGS="${CPPFLAGS} -I/mingw/include"
export CXXFLAGS="${CXXFLAGS} -I/mingw/include"
export LDFLAGS="${LDFLAGS} -L/mingw/lib"
export JAVA_HOME=/share/ProgramFiles/java/jdk1.7.0_04
export JRE_HOME=/share/ProgramFiles/java/jre6
export CLASSPATH=.
EIO
 

TODO:

Everyone else, I'm going to have to hold off on delivering an rpm.sh script until I'm done building CLFS on my windows computer. I apologize. Fortunately, since it's part of the GNU/Linux specification, if you're on a linux computer you've got rpm. And there are ports out there to most other unix-likes.

 

Using RPM

So now that we have RPM built, let's go over quickly how to use it.

First, if you haven't already done so, download the seed.spec file and put it in your specs folder. You'll notice this looks a LOT like the sh scripts we've already been using. That was on purpose. RPM uses something called a "spec" file to build packages from source. Doing things with RPM gives us a little more control over the build process, and gives us some options when things go wrong and packages fail to build. And, probably most importantly, it packages everything up into something that is very easy to uninstall or install directly on another computer.

Let's modify that file so that works for beecrypt. It may be best for learning purposes if you tried to modify it by hand first. However, this is what the finished product should look like:

file beecrypt.spec:

Summary: Beecrypt cryptography toolkit
Name: beecrypt
Version: 4.2.1
Release: 1
%define TARFMT tar.gz
Source0: http://download.sourceforge.net/%{name}/%{name}-%{version}.%{TARFMT}
Patch0: %{_patchdir}/%{name}-%{version}-mingw_make_dlls.patch
License: LGPL
URL: http://beecrypt.sourceforge.net/
%define MD5SUM 8441c014170823f2dff97e33df55af1e
Group: Development/Tools/Building
BuildRoot: %{_builddir}/%{name}-root
%description
"BeeCrypt is an ongoing project to provide a strong
and fast cryptography toolkit. Includes entropy sources,
random generators, block ciphers, hash functions,
message authentication codes, multiprecision integer
routines, and public key primitives.

Optional Depedencies:
GCC (for GCJ Java Compiler) or JDK
Python

Contents:
Installed Directories: /usr/include/beecrypt
Installed Libraries:   libbeecrypt.{a,so}

Short Descriptions
libbeecrypt.{a,so}:    a library that contains functions
                       for strong and fast cryptography."


%prep
%setup -q -n %{name}-%{version}
%patch0 -p1
test "$(/usr/bin/md5sum %{SOURCE0} | awk '{print $1;}')" == "%{MD5SUM}"

%build
cat > used_config << EOF
# Doing things this way, we can add comments here
--prefix=%{_installprefix}
--sysconfdir=%{_sysconfdir}
EOF
./configure $(sed '/^#/d' used_config)

# Omit the -jn option if you don't have multiple processors or a multi-core,
#   or just if a package is giving you trouble.
# Otherwise, it might help your builds go faster if you set this to
#   (number of processors) x 2 + 1 (e.g. this was built on a dual-core)
CFLAGS="%{_platformincludes}" LDFLAGS="%{_platformlibs}" make -j3

%install
mkdir -pv $RPM_BUILD_ROOT
make DESTDIR=$RPM_BUILD_ROOT install

%clean
#rm -rvf %{_builddir}/*

%files
%defattr(-,root,root)
/

 

As you can see, this is very similar to the beecrypt.sh we had up earlier.

 

Summary: Beecrypt ...
...
Buildroot: ...

This section is the header. One thing to note: if you set the source location to a url as we've done, RPM will download the source for you if you don't have it already on some systems. Unfortunately, this is an optional feature that isn't always compiled into RPM, so you may have to download beecrypt again manually if you got rid of it earlier. Otherwise, all these values were copied in from the beecrypt.sh header.

 

%description
...

We just copied the description we used earlier, minus the URL (which went in the header) and the line "Introduction to Beecrypt"

 

%prep
%setup -q -n %{name}-%{version}
%patch0 -p1
test "$(/usr/bin/md5sum %{SOURCE0} | awk '{print $1;}')" == "%{MD5SUM}"

This section does a few things.

First, it checks to see if you've got the source available to you, as well as any patches you specified. Note that on some systems (I can confirm mingw), it won't respect your defines and will insist on looking inside your source directory for patches. It won't on others. So you may have to copy the contents of your patch directory back into your sources directory.

Next, it untars your package into your build directory. It then changes into the directory you specified. The value I specified here, packname-version, is the default, but I specify it manually anyway to make changes faster for future packages where we need to specify something else.

Next, it applies the patch we're using.

Finally, it runs the md5 sum test to authenticate the package. RPM runs everything with "set -e", so if this test fails, rpm exits immediately.

 

%build
...

This section was, again, copied from the beecrypt.sh script. I made a small change in the way I'm running the configure script to allow you to comment things out in used_config.txt

 

%install
...

Again, copied from beecrypt.sh.

 

%clean
...

If you uncomment this section, rpm will automatically remove the build directory and the temporary install directory it creates for packaging.

 

%files
...

If you only wanted some of the files you installed to go into your package, you can fine tune things down here. By using /, we are telling it it's okay to include everything in the package we make.

 

Now we need one more piece in place before any of that will work. In the scripts directory, if you haven't grabed it already, download this file and then copy it into your home directory:

cd ~/clfs/scripts
wget http://greenback.gremlin.net/RENAME.rpmmacros
cp -v RENAME.rpmmacros ~/.rpmmacros

What this did:

You noticed all those confusing %{_something}s scattered throughout the RPM script? You probably guessed those were names that stood for something else, and if you did, you guessed right. Go ahead and look inside RENAME.rpmmacros. You can define these however you see fit. If you're not on MinGW, you'll definitely need to uncomment some lines and comment others as instructed. When you're done editing RENAME.rpmmacros, go ahead and copy it back over to ~/.rpmmacros again.

Alright, we're finally ready to give building beecrypt a whirl. To complete this task, run the following commands:

rpmbuild -ba ~/clfs/specs/beecrypt.spec