Cross Linux From Scratch

This course will become read-only in the near future. Tell us at community.p2pu.org if that is a problem.

RPM [June 15, 2012, 3 a.m.]


Since this package has a lot of little dependencies, I decided to throw in a quick table of contents. If you know you have everything, just go ahead and jump right on down to the RPM section.

Why Package? Aren't we using Git?

Even if you don't want to use RPM to package with, you might want to read through this task. The strategies I'll be going over can be used with any source-based package management. 

In addition to version control, you'll also want some sort of package management. Packaging offers a few advantages simple version control doesn't:

  • It's very easy to build a package once, and then install it on other computers, without having to rebuild
     
  • If you are compiling a long line of packages, and you mess up on package 3 which 6 and 7 depend on but 4 and 5 don't, you can uninstall 3, 6, and 7 without affecting 4 and 5. This would be very difficult using version control alone.
     
  • It's very easy to make minor changes to packages if you have to reinstall them later, such as compiling in a new feature
     
  • It's very easy to upgrade packages to newer versions without destabilizing your whole OS
     
  • Finally, just about any package management system you can use (including simple shell scripts; check out Sorcery and Sourcemage some time, or my personal favorite source-based package repository, slackbuilds.org) probably has a community with pre-written source packages and even pre-compiled binaries. Often, installing software on Linux can be just as easy as installing software on windows, and as an added bonus you have more control and don't feel dirty signing any EULA contracts. Packaging means Linux can be "double-click and run" too.

 

Packaging and version control complement each other well. Version control makes it very easy to keep track of configurations and metadata. If you changed something in your spec (build instructions) to work with a specific version of a package, for example, or you want to use an old version of a startup script, version control is a very good tool for this purpose. And if you don't want to have to repeat a lot of steps creating software, packaging is a good tool to use here. With both combined, you'll almost never have to worry about things being deleted, and your system will almost always be recoverable. And best of all, you won't need hundreds of Gigabytes to back up your OS; your OS will constantly be backed up as you use it.

If you haven't picked up on it by now, I'm also a huge fan of a third kind of backing up: Read-only media. Git and RPM work great for solving just about any software pickle you can find yourself in. But if your hard disk itself ever gets damaged, or you accidentally do something catastrophic to your software, then they'll fall short. The reason why CDs and DVDs are great, is that you write to them once, and then you can't overwrite them. And it's really easy to port stuff over to other computers without setting up a network or using Dropbox.

Another great solution is "cloud storage". If you send your data over a network to another computer (including the servers housing your e-mail or Dropbox stuff), that's what's called "write restricted". It means that there are enough steps in place that you really have to make an effort to permanently delete something. With e-mail and Dropbox, this also serves as a kind of version control as well. With Linux, it's very easy to have total control over every single computer your data touches, so you don't even have the security and privacy concerns you'd have using e-mail.

Since the CLFS and CBLFS books are, in a sense, a source-based package management system in and of themselves, I'd highly recommend going that route. The difference between source-based package management and binary-based package management, is that with source-based packaging, you download and build the package from source fresh every time you want to install it, and with binary-based packaging, you just download a package you already built and install that instead. Source-based is more flexible and you don't get bogged down with dependencies. Binary-based is faster and easier. Windows setup.exes are an example of binary-based package management.

You are free, of course, to use a binary-based package management system if you want. That's going to be a little more in depth than anything I'll cover here, but the gist of it is that you'll want to write your own software that uses a database of some kind (probably SQLite) to keep track of compatibilities. Git is great for distributing your packages online once you've done that and keeping track of dependency sets. I'm a lighttpd fan, but the good old Apache httpd server has a lot of built-in support for git. I won't be showing you how to modify the database that RPM itself uses to track binary packages and dependencies because, frankly, I don't know how and don't know where to find this information online. I've tried tracking it down in the past unsuccessfully. So I'll be installing everything with the --nodeps flag as a result.

Alrighty, I'm done yappin. Let's get to work.

Beecrypt

RPM has one dependency, Beecrypt. Beecrypt and RPM have CBLFS pages here:

http://cblfs.cross-lfs.org/index.php/Beecrypt

http://cblfs.cross-lfs.org/index.php/RPM5

One other dependency you'll want for beecrypt is JDK. Make sure JAVA_HOME is set, and $JAVA_HOME/bin is on your path. Here's a guide on doing that:

http://www3.ntu.edu.sg/home/ehchua/programming/howto/Environment_Variables.html

To get this to build on MinGW/MSYS, I had to patch a file. I found out how to patch the file by going to the project's homepage, which I found on the project's CBLFS page.The patch was designed not to interfere with your build process in any environment, but was designed with MinGW in mind. The patch can downloaded from here, Since I'll be going over it later, I'll also paste it right into these instructions.

file beecrypt-4.2.1-mingw_make_dlls.patch :

--- beecrypt-4.2.1/md4.c.orig    2012-06-01 18:33:53 -0500
+++ beecrypt-4.2.1/md4.c         2012-06-01 18:35:47 -0500
@@ -1,4 +1,18 @@
 /*
+http://sourceforge.net/projects/beecrypt/forums/forum/27737/topic/3351667
+
+Developer suggested bug fix for the following problem:
+
+.libs/libbeecrypt.dll.a.libs/md4.o:md4.c:(.text+0x571):
+ undefined reference to \`_imp__mpsetw'
+
+Enable by setting -DMINGW_MAKEDLLS in CFLAGS
+*/
+#ifdef MINGW_MAKEDLLS
+#define BEECRYPT_DLL_EXPORT
+#endif /* MINGW_MAKEDLLS */
+
+/*
  *
  */

 

To create the patch, I followed this four step process

  1. Copy the file you're changing to the filename plus .orig

    cp md4.c{,.orig}
     
  2. Edit the file you're changing

    vim md4.c
     
  3. Change to your build directory

    cd ~/clfs/build
     
  4. Run the diff tool on the original and changed files  to create the patch

    diff -du beecrypt-4.2.1/md4.c{.orig,} > \
      ~/clfs/patches/beecrypt-4.2.1-mingw_make_dlls.patch

 

Note that the comma was before .orig the first time and after .orig the second time. Also note the naming convention I used for the patch; it's packagename-version-description.patch.

To learn more about patches, go ahead and open the patch in a text editor. It contains a URL. Go ahead and open that URL too. You'll notice I was patching a c file. You don't have to be a full-fledged c programmer to benefit from this information, but I wanted to go ahead and walk you through the changes I made.

First, every time you see a line starting with

+

(your browser might be wrapping long lines, but most of them start with a +) that means that it's something that is in the changed file and not the original - in other words, something that was added. If something was removed, that would mean starting with a

-

Go ahead and ignore the + or - starting the lines above now that you know what they mean.

Second, everything between
/*

and

*/

is commented out. Just like using # in bash, the c compiler ignores everything between /* and */. I added some helpful information in-between those tags in case someone got hold of my patch and wanted to learn more about it.

Third,

#ifndef MINGW_MAKEDLLS

and

#endif

means in plain english, "only do this stuff if we're using MinGW". It pulls this off by first checking to see if something called MINGW_MAKEDLLS exists. Later I'll show you how I set this, but it's enough to know right now that it only exists if we're really using MinGW. I put this in there so that the patch wouldn't interfere with the linux/mac/bsd people.

Finally,

#define BEECRYPT_DLL_EXPORT

is just word-for-word what they told me to add in the online help.

Our shell script for building beecrypt, which you can download here, is as you'll notice a little more complicated:

file beecrypt.sh:

#!/bin/bash

# Exit if anything goes wrong immediately
set -e

# These should stay the same between packages
BUILDDIR="/home/sean/clfs/build"
PATCHDIR="/home/sean/clfs/patches"
SOURCEDIR="/home/sean/clfs/sources"
PREFIX="/usr"
SYSCONFDIR="/etc"
CUID="${UID}"
if [ -f /msys.bat ]
then
  PREFIX="/mingw"
  net localgroup Administrators | grep "`whoami`" && CUID="0"
fi

# These change with every package
PACKNAME="beecrypt"
VERSION="4.2.1"
TARFMT="tar.gz"
TARBALL="${PACKNAME}-${VERSION}.${TARFMT}"
PACKDIR="${PACKNAME}-${VERSION}"
SOURCE="http://downloads.sourceforge.net/${PACKNAME}/${TARBALL}"
MD5SUM="8441c014170823f2dff97e33df55af1e"

# Specify options here
export CFLAGS="-I${JAVA_HOME}/include"
export CPPFLAGS="${CFLAGS}"

# Set compatibility options here
if [ -f /msys.bat ]
then
  export CFLAGS="${CFLAGS} -I${JAVA_HOME}/include/win32 -DMINGW_MAKEDLLS"
  export CPPFLAGS="${CFLAGS}"
  export LDFLAGS="-lgomp"
fi

# Description
cat << "EIO"
"Introduction to Beecrypt

BeeCrypt is an ongoing project to provide a strong
 and fast cryptography toolkit. Includes entropy sources,
 random generators, block ciphers, hash functions,
 message authentication codes, multiprecision integer
 routines, and public key primitives.

Project Homepage: http://beecrypt.sourceforge.net/
Optional Depdencies:
  GCC (for GCJ Java Compiler) or JDK
  Python

Contents:
  Installed Directories: /usr/include/beecrypt
  Installed Libraries:   libbeecrypt.{a,so}

Short Descriptions
  libbeecrypt.{a,so}: a library that contains functions
                      for strong and fast cryptography."
EIO

if [ "${CUID}" != "0" ]
then
  # Download the source package
  cd $SOURCEDIR
  [ ! -f $SOURCEDIR/$TARBALL ] && wget $SOURCE
  [ "`md5sum ${TARBALL} | awk '{print $1;}'`" != "${MD5SUM}" ] \
    && echo "Error: this doesn't look like the right package" \
    && exit 1

  # Extract the source
  cd $BUILDDIR
  rm -rf $PACKDIR # We always want to insist on a clean build
  tar xvf $SOURCEDIR/$TARBALL
  cd $PACKDIR

  # Patch
  patch -Np1 -i "${PATCHDIR}/${PACKDIR}-mingw_make_dlls.patch"

  # Configure and compile the package:
  cat > used_config.txt << EIO
--prefix=$PREFIX
--sysconfdir=$SYSCONFDIR
EIO
  ./configure "`cat used_config.txt`"
  make

  echo "Successfully built, run this script as root to install"

# Install the package:
elif [ ! -d "${BUILDDIR}/${PACKDIR}" ]
then
  echo "Run this as an ordinary user first"
  exit 1
else
  cd "${BUILDDIR}/${PACKDIR}"

  # Run any tests

  # Install the package
  make install

  # Install documentation

fi

Note: I made some cosmetic changes to the above script to aid the wiki's built-in syntax highlighter. Both the above script and the one you download do exactly the same thing.

This is done so that we can copy and paste this whole script to create new packages while only changing a few lines. I'll go ahead and walk you through the sections really quickly.

# These should stay the same between packages

Everything in this section you can edit one time, to change "sean" to your username, and then never touch again no matter how many times you copy it. It's cross platform (although let me know if there's a better way to check to see if you're inside the MinGW/MSYS environment in the comments section below).

# These change with every package

This is where we take advantage of some common conventions. Usually with a unix source package, you'll download something that looks like "beecrypt-4.2.1.tar.gz".

  • The first part, "beecrypt", is the package name, which we'll call PACKNAME
  • The second part, "4.2.1", is the version, which we'll call VERSION
  • The third part, "tar.gz", is the compression format, and might also be "tar.bz2", "tar.xz", or rarely, "tgz" or "zip". We'll call this TARFMT.

 

Usually this extracts to a folder called "PACKNAME-VERSION" (so in my case, "beecrypt-4.2.1"). We set these here not just to save us keystrokes later, but to avoid typos. If you get it right here, you get it right everywhere you use PACKNAME. It's also right at the beginning, so it's easy to change to whatever package we're installing.

We also have something called "MD5SUM" here. This is what we call an "integrity check", which is just a way to check to make sure the source you downloaded is really the right source. I got this by first manually downloading the source I wanted to use, and then running this on it:

md5sum $PACKNAME-$VERSION.$TARFMT | awk '{print $1;}'

This comes in really handy to prevent, for instance, you accidentally grabbing the wrong version of a file.

# Specify options here

For most packages, you can leave these lines blank. If you need to set CFLAGS, CXXFLAGS, LDFLAGS, or any other environment variables, you can set them here so they're close to the top of the file. You'll see this happen from time to time in the CLFS book, especially early on. In this case, we need them to include java headers.

# Set compatibility options here

Another one that can usually be left blank. I set stuff here so that the script would be run a different way if you were running MinGW. Incidentally, remember earlier how I said I'd show you how I defined MINGW_MAKE_DLLS only if we're using MinGW? This is where I did that, and now you know :)

# Description
cat << EIO
...
EIO

I copied and pasted the description from the CBLFS page right in here in place of the ... . This is completely optional. It's nice for me because I don't have to go to the CBLFS page if I want to know what beecrypt does later.


if [ "${CUID}" != "0" ]
then
  ...

This causes the script to run a different way if you're root, than it does if you're an ordinary user. It's useful if, like we'll be doing here, you're building packages as an ordinary user and installing them as root.

  # Download the source package

Exactly "what it says on the tin". Note that we only download the source package if we don't already have it. Also, we'll go ahead and run that integrity check here. As you can see, we use the same command to run the integrity check as we did to get MD5SUM to begin with. So if you forget, it's right here. If anything goes wrong, we bail out right here.

  # Extract the source
  ...
  # Patch
  ...
  # Configure and compile the package
  ...
 

We copy the instructions from the CBLFS page in here, with a few tweaks.

First, you'll notice I have everything organized into three sections. It's okay if you don't do this. I did this to make the transition to RPM specs a little more smooth later on.

Second, I adding the patch. One thing you might have to tweak here from time to time, is changing -Np1 to -Np0 or some other number. For us here we won't need to change it, but not everyone follows the convention of patching from an outer directory like we did.

Last, I also dumped all the configure options to a text file. This is an extremely useful practice if a package fails to build and you're experimenting with configure flags, and I'd highly encourage you to do this. The cat tool combined with the backticks will compress everything onto a single line for you automatically, so you don't have to worry about ending lines with \.

Again, if you want to just get something working, it's okay to copy instructions from the CLFS or CBLFS book in this section verbatim and ignore all the formatting and categorizing. It's up to you.

  echo "Successfully built, run this script as root to install

# Install the package:
elif [ ! -d $BUILDDIR/$PACKDIR ]
then
  echo "Run this as an ordinary user first"
  exit 1
else
  cd $BUILDDIR/$PACKDIR

  # Run any tests

  # Install the package
  make install

  # Install documentation

fi

First, we let the ordinary user know that everything worked out okay and we're ready to install. Next, we throw in a quick check to make sure we didn't try running this script as root before running it as an ordinary user. Finally, we copy whatever installation instructions there are in the CLFS or CBLFS book page (usually just "make install") in. I left a spot in there for you to run things like "make check" if you want to. Sometimes there are also special installation instructions for installing documentation. Here there aren't any.

That's it for beecrypt. If anything went wrong for you, please refer to the task for building git.

Popt

Even though this is listen in the CBLFS page as being "recommended", some systems will require you to have popt before RPM will install.

For unix-likes, Popt's CBLFS page is here:

http://cblfs.cross-lfs.org/index.php/Popt 

I will have a popt.sh script ready in the future, but it shouldn't be too hard to write one in the meantime. You can also just use your package manager to install it. Chances are you already have it; look for /usr/include/popt.h. If it's there, you have popt.

For windows, you'll want to grab popt-1.8-1-lib from the GnuWin32 site, here:

http://sourceforge.net/projects/gnuwin32/files/popt/1.8-1/popt-1.8-1-lib.zip/download

The next step on windows is to, in an admin shell, do this (replace sean with your username)

cd /mingw
unzip /home/sean/clfs/sources/popt-1.8-1-lib.zip

GNULib

If you're using Linux, a Mac, or BSD, you can skip right down to installing RPM, but if the build process fails, come back to this section. RPM requires a full POSIX environment to build . Some environments don't have the full POSIX specification. MinGW is one such environment, and although it's nearly complete, Cygwin is another one. In general, if you're on a system missing some POSIX headers and libraries, when you run the make utility, it might crash wiith a message like

fatal error: sys/aio.h:  no such file or directory

If this happens to you, this section is for you.

When we need to compile a package in an environment it wasn't designed for, that's called "porting". The best way to port is to use a compatibility library. Here, we'll use a great one called GNULib.

http://www.gnu.org/software/gnulib/manual/gnulib.html

Let's go ahead and grab it. Make sure git is on your path, and if it isn't, add the following line to ~/.profile and reload your shell (change it as needed if you didn't do the windows task earlier)

export PATH="${PATH}:/share/ProgramFiles/Git/bin"

Then run these commands:

cd ~/clfs
git clone git://git.savannah.gnu/gnulib.git

Note that we did this outside of our build directory, and put it right in our nest. This is because we'll be using a tool, creatively named gnulib-tool, to pull in gnulib headers and libraries.  From now on, gnulib will be a permanent part of ~/clfs nest. So we'll go ahead and add this line to our ~/.profile next to the other PATH exports:

export PATH="${PATH}:${HOME}/clfs/gnulib"

That's all there is to installing gnulib. Now I'll show you how to use it. You won't have to do this yourself, this is just for reference. You can skip this now if you want and go right to "RPM".

  1. Go here,

    http://www.gnu.org/software/gnulib/manual/gnulib.html

    and look in section 7 for the header you are missing.
     
  2. Click the link for that header. Examine the "Portability problems fixed by Gnulib" section. If your problem is listed under this, and NOT listed under "Portability problems not fixed by Gnulib", then proceed to step 3. Otherwise, you can't use Gnulib to solve your problem.
     
  3. Run these commands. I am going to assume you are missing sys/aio.h and netdb.h. sys/aio.h becomes sys_aio and netdb.h becomes netdb. If you'd rather not guess at the package name that matches your header, you can look at the link in step 1. Also, you'll need to change your host if you aren't using MinGW.

    cd ~/clfs/build
    rm -rf gl
    gnulib-tool --dir gl --create-testdir sys_aio netdb
    cd gl
    ./configure --host=i686-pc-mingw32
    make
     
  4. You now have a compatibility header. You can use it with the following:

    export CFLAGS="${CFLAGS} -I${HOME}/clfs/build/gl/gllibs"
    export CPPFLAGS="${CPPFLAGS} -I${HOME}/clfs/build/gl/gllibs"
    export CXXFLAGS="${CXXFLAGS} -I${HOME}/clfs/build/gl/gllibs"

    Note here that "PP" stands for "Pre-Processor", not "Plus-Plus". CXXFLAGS is used by the C++ compiler.

I'll be making a function in the RPM script to do all that automatically for us.

 

RPM

Alright, finally ready to build rpm. Here's the spec:

http://greenback.gremlin.net/rpm.sh

This is the most complex script yet, but the additions are pretty simple and easy to go over.

function gnulibport()
{
  ...
}

This is a function, which is like a script-within-a-script. When you call it like this:

gnulibport sys/uio.h

it adds sys/uio.h into a queue of headers to build. It stores this queue in the environment variable GNULIB_HEADS. When the queue is full and you're ready to build those headers, you call it like this:

gnulibport make

Then there's the second function:

function testheader
{
  ...
}

The traditional way to test to see if you have a c or c++ header is to make a short program and try to build it; if it fails to build, then it's assumed it's because you were missing a header. However, I wanted this tutorial to be accessible to people that aren't programmers. So instead, this function just looks in all your include directories.

It may seem backwards, but if it finds anything, it "returns" 0, and if it fails to find anything, it "returns" 1. "Returning" means that when the program exits, it sets an environment variable to a number. That environment variable is:

$?

When that environment variable is set to 0, that means "everything is ok". Try it out yourself with the programs /bin/true and /bin/false.

I use both these functions in this section:

# We're going to need some headers that aren't on all systems

so you can see them in action. Finally, you'll notice there's two more directories this time: SYSCONFDIR and LOCALSTATEDIR. They get set to /etc, which is where configuration files you as the user are supposed to edit to change how programs work go, and /var, which is like scratch space for your program. It uses this to keep track of it's "state", which means things like whether the program is running or the last thing the program was doing when it was shut off last. It also puts logs in /var.

 

Using RPM

So now that we have RPM build, let's go over quickly how to use it. First, if you haven't already done so, download the seed.spec file and put it in your specs folder. You'll notice this looks a LOT like the sh scripts we've already been using. That was on purpose. RPM uses something called a "spec" file to build packages from source. Doing things with RPM gives us a little more control over the build process, and gives us some options when things go wrong and packages fail to build. And, probably most importantly, it packages everything up into something that is very easy to uninstall or install directly on another computer.

Your homework, to complete this task, is to turn beecrypt.sh and rpm.sh into beecrypt.spec and rpm.spec . It's enough to write them out, but for extra credit you should run them and try to get them to build. I'll go over how to do that in just a second. First, though, let's go over the sections in the spec to make things easier for you.

First, we have the variables up top: