Cross Linux From Scratch

This course will become read-only in the near future. Tell us at community.p2pu.org if that is a problem.

Windows Environment Set-Up [June 1, 2012, 2:53 p.m.]


Cross linux from scratch can be built from just about any operating system with a working C compiler.  I'll walk you through a Windows, MinGW/MSYS environment setup. It's probably easiest if you're having trouble, though, to skip this task and do this from a BSD, Mac, or a Linux computer. Linux has several LiveCD, LiveUSB, and virtualized distributions that are worth checking out. I think Linux from Scratch even has their own LiveCD made just for us. Any of these will have working c compilers and environments that will suit our needs.

I made a quick video in three parts weighing in at about half an hour helping you if you're starting out the Windows way. You might want to full screen. If you're not doing this on windows, you can skip down to the next section.

http://youtu.be/U2Agy6ulJtc

One thing I neglected to mention (since I didn't realize I needed it until later), you'll also want to get the "xz" and "mingw-developer-toolkit" packages with mingw-get, in an admin shell, like this:

mingw-get install xz
mingw-get install mingw-developer-toolkit

http://youtu.be/it4kGicvADI

I didn't get around to it in the next video, but you can chain configure and make together with a double-ampersand like

./configure --all-your-config-options && make

http://youtu.be/-Tl2WsGfw1M

 

Ext2 File System

We'll be building CLFS on an ext2 filesystem. Windows doesn't have to be able to access ext2 directly, but I'll show you how if you'd like to be able to. You will need to be able to create and populate ext2 filesystems, though, and I'll show you how to do that as well.

A filesystem is a way for an operating system to store files, folders, and information about those files and folders such as security permissions and when a file was last accessed. Windows has two native filesystems, FAT and NTFS. Linux can work with both of these, but ext2 is still preferable. The advantage to using ext2 is that just about any disk repair and boot loader utility can read it. This means that if anything ever goes wrong with your computer, you can always download a LiveCD and fix any problem with your disk itself (except maybe fire and explosion ;) ). It's also easier to use Linux's native disk maintenance utilities with ext2.

If you want to be able to use ext2 drives or partitions natively from windows, there's two options available to you. I've had more luck with Ext2FsD than Ext2IFS on Windows 7 64 bit personally. If neither of these work for you you can still build CLFS on an ext2 filesystem from windows, you just won't be able to access the OS you build directly from windows. Remember to install Ext2IFS in compatibility mode for Windows Vista Service Pack 2 if you're on windows 7.

http://www.fs-driver.org/

http://www.ext2fsd.com/

Even if neither of those are installed, you can create and populate an ext2 filesystem from windows using Cygwin. You can grab cygwin from here:

http://cygwin.com/install.html

Cygwin installation is a little involved for this tutorial (which is why I linked to the install help page instead of the download page), but make sure that these two things are true:

  1. You've installed cygwin to a different top-level directory from mingw/msys (in other words, something like C:\cygwin and C:\mingw would work)
  2. You've installed the e2fsprogs and e2fsimage packages, found under "System"

Here's a short video tutorial that will guide you through everything once you have all that set up.

Caution

Please note that, although you should be safe if you follow the instructions in the video, it contains a potentially unsafe command. The tool "dd" should always be used with care. Although it originally meant something like "duplicate disk", It has a nickname in the linux community: "destroy disk". The potentially unsafe command is optional, but you can't make a bootable OS on a USB stick without it. If you follow the instructions in the video you should be fine, the risk is minimal. However, this would be great place for me to recommend you back up any important files and have a recovery disk for your OS handy.

<video is in the process if being replaced>

 

Git and the Generic 6 Step Package Build Process

While you will want a git binary available to you, compiling from source in this section is entirely optional. Depending on how "from scratch" you'd like to be, there are precompiled or pre-packaged versions of git available to just about every operating system. Git hosts downloads here,

http://git-scm.com/downloads

Windows users only

First, when you install the binary you get from the git website, you'll want to make sure that the "Run git from the Windows Command Prompt" radio button is selected, followed by the "Checkout as-is, commit Unix-style line endings" button on the next screen.

Second, this time, it's you MinGW/MSYS that catch a break and get to skip ahead to the next section. You are welcome to try to build git on windows, but I was unsuccessful under MinGW/MSYS. I'm going to use the pre-compiled binary. If you want to try, one thing I had to do was create a config.cache file with the following contents:

git_cv_socklen_t_equiv=int

and then add the configure option,

--cache-file=config.cache

However, my 64 bit windows would still not include winsock2.h correctly. You're also going to need zlib, which you can install using mingw-get.

Otherwise, there is a great development community specifically for porting git to windows, including building it from source, here

http://code.google.com/p/msysgit/

and if you're comfortable trying your luck with cygwin instead of msys (bearing in mind that your git can only be used in your cygwin environment and not your msys environment), there's a tutorial I haven't tried here,

http://techblogging.wordpress.com/2008/04/11/compiling-and-installing-git-on-windows-under-cygwin/

What you can do is read through this section for generic package building instructions, just realize that it's not going to apply directly to you. The git download page I linked earlier has a link to the windows binary, and I've provided a direct link here as well:

https://github.com/downloads/schacon/testy/Git-1.7.10-preview20120409.exe

Everyone else

So if you're still reading this section, you're in a linux, BSD, or mac environment, or you're just here to learn.

There's more flavor text in the next paragraph, feel free to skip it.

One of the biggest complaints people have building LFS is that it's frustrating when a package doesn't build correctly, and you have to scrap everything and start over. Some people may have joked that it would be a lot easier with a time machine. Well guess what! Git gives us a kind of a "time machine". You can undo and redo operations, create multiple timelines, merge them back together, cherry pick and ressurect anything you've done in the past, and resolve conflicting visions between you and your fellow time travelers, among other things. It has a little bit of a learning curve to use it to its full potential, but we'll only be using very simple commands.

Feel free to just install git the easy way and skip to the next section if you have any problems here. My rational for building for source here, is to A.) ensure that our build environment works, and b.) give you a gentle walk-through of the unix packaging process. We're going to create what's called a "source controlled package". This section and the troubleshooting section will be a reference tool for you later if you have any problems.

We're going to use the page from the "CBLFS" book to build this package. The "B" stands for "beyond". It's appropriate, then, that we're pulling the instructions for creating our time machine from the future, in the book intended to be used once the Cross Linux From Scratch system has already been built.

Enough talk. Allons-Y!

The CBLFS page for git is here:

http://cblfs.cross-lfs.org/index.php/GIT

By the way, that's also an easy way to look up generic build instructions for any linux package ever; just type in

http://cblfs.cross-lfs.org/index.php/<name_of_package>

The first letter is usually capitalized, and spaces are underscores. If you don't find the package that way (I didn't in this case because GIT is all capitals), just use the search box in the lower left hand corner. If you come across a rare package that isn't up there already, and you figure out how to build and install it, consider making a CBLFS page yourself.

Step 1: Grab Sources

According to the CBLFS book page, the sources are located at

http://git-core.googlecode.com/files/git-1.7.8.2.tar.gz

http://git-core.googlecode.com/files/git-manpages-1.7.8.2.tar.gz

We'll go ahead and grab those.

If it feels like I'm talking a little fast at this point, this would be a good place to review the links in the "prerequisites" section. Things are going to get a little technical from here on out.

First, fire up a shell as an ordinary user. We'll want to be an ordinary user for steps 1-5. Then:


cd ~/clfs/sources
wget http://git-core.googlecode.com/files/git-1.7.8.2.tar.gz
wget http://git-core.googlecode.com/files/git-manpages-1.7.8.2.tar.gz

That's going to be step 1 whenever we grab anything from the CBLFS book, by the way; grab all the sources listed at the top of the page, and dump them in our clfs/sources folder.

Step 2: Check Dependencies

Step 2 is going to be looking at the dependencies section. In this case, there's 2 required packages:

  • Curl
  • Expat
     

You'll want to use your OS's standard package management tools to get those. If you're reading this and you've been following instructions, you've already got expat. If you want to try your luck building Curl from scratch, here's its CBLFS page:

http://cblfs.cross-lfs.org/index.php/Curl

If you're doing things the hard way (go you!), the build instructions for Curl are going to be nearly identical to the ones for git, so just keep reading. if you need help building packages from source, you can use the second video in the Environment section for help. There, I build the ncurses package from source on windows, but the same general procedure should work for you too.

Step 3: Write a spec

Once you've got all your dependencies installed, step 3 is to copy the build instructions into a packager script. Since we don't have RPM installed yet, we're going to make a shell script for this. I'm doing things this way in case you decide you'd rather not use RPM, so that you'll have some way of packaging. You can always use shell scripts for packaging no matter what.

For this tutorial I'll be building a non-multilib system. As a challenge to yourself, if your processor can handle 64 bit libraries, you can go back and build a multilib system later. Let's go to the git page, highlight all the instructions under "non multilib", and copy them. Then we'll do this in our shell:

cd ~/clfs/specs
cat > git.sh << "EIEIO"
Compile the package:
./configure --prefix=/usr --libexecdir=/usr/lib \
    --sysconfdir=/etc &&
make
Install the package
make install
Install the man pages:
tar xvf ../git-manpages-1.7.8.2.tar.gz -C /usr/share/man
EIEIO

Note that you don't have to type in all of that; everything between the EIEIOs I just pasted in.

Next, let's open that file we just made (it's called "git.sh") in our favorite text editor. My personal favorite is vim. Other good command line choices are emacs, nano, and joe. If you open in a graphical editor, make sure there's no text formatting going on. If you've got KDE, I like kate.

#!/bin/bash

set -e

# This should be /usr for bsd, mac, or linux, or /mingw on windows
PREFIX=/usr

#Compile the package:
./configure --prefix=$PREFIX --libexecdir=$PREFIX/lib \
  --sysconfdir=/etc && make

# Install the package
make install

# Install the man pages:
tar xvf ../git-manpages-1.7.8.2.tar.gz -C /usr/share/man

What we did there is:

  • added the line "#!/bin/bash" at the beginning


    • The "#!/bin/bash" line is called the "shebang". It says that when we run the script as an executable, we should use /bin/bash to interpret it. So if this were a python script instead, we'd have something like "#!/usr/bin/python".
       
  • added the line "set -e"


    • "set -e" causes the script to exit immediately if something goes wrong.
       
  • added a few "#" marks at the beginnings of a few lines.


    • These are what are known as "commenting out a line". The bash interpreter, /bin/bash, ignores everything after one of those hash marks. If you look at the git CBLFS page, you'll notice I put one of those marks everywhere the line was a comment not meant to be run, so that only the real commands get executed. I like leaving those in there, because they keep the script real easy to read if I have to make changes later.
       
  • Set up a PREFIX environment variable, and changed --prefix=/usr to --prefix=$PREFIX


    • This is the directory you're installing to. The entire directory tree for git is going to be under this prefix. So binaries will go into $PREFIX/bin, c and c++ headers will go into $PREFIX/include, and libraries will go into $PREFIX/lib. Sometimes packages need other configuration files. Those go into folders called etc, var, and shared. We'll go over those later on when we start build CLFS. I left some suggestions as to what PREFIX should be in the comment I added.


      • ADVANCED: One other suggestion, if you're trying to do everything as an ordinary user, is to change this directory to one you have write permissions on as an ordinary user. Just be sure you know what you're doing. This is a little advanced for this tutorial, but if you go this route, you'll need to set your PATH up so that your binaries are on it, and you'll need to setup your CFLAGS variable to include -I${PREFIX}/include and your LDFLAGS variable to include -L${PREFIX}/include. Those shouldn't be substitutions, by the way; go ahead and expand them because you'll be setting PREFIX to other things.

We're going to need to make one more quick change to that last line. If you recall, our git-manpages tarball is inside our sources directory. We also (probably) don't have permission to install anything as an ordinary user, so we'll just echo the command so that we can copy/paste it into an admin shell later. For quick changes, I like to use the sed tool instead of firing up my editor. We can edit everything like this:

sed -e 's@\.\./git-manpages@~/clfs/sources/git-manpages@' \
    -e 's/^tar.\+/echo "&"/' \
    -e 's/make install/echo "&"/' \
    -i git.sh

The sed tool relies on something called "regular expressions".
You can look those up online if you'd like to learn how to use the sed tool for yourself. 

Step 4: Extract the package's build directory

Okay. Now that we've prepared our installer script, which I'll call "spec" from now on, it's on to step 4. Step 4 is changing to our build directory and extracting.

cd ~/clfs/build
tar xvf ../sources/git-1.7.8.2.tar.gz
cd git-1.7.8.2

Easy, right?

Step 5: Build the package

Step 5 is the one most likely to give you headaches. So if anything goes wrong, the entire troubleshooting section below is dedicated just to step 5 of the build process here. I've bolded out the stuff that's most likely to go wrong to make it easier to find.

All that buildup, you ready for step 5?

sh ../../specs/git.sh

Step 6: Installation

Step 6, is to install everything we just built. If you were following instructions earlier, you're an ordinary user and don't have the authority to install anything. That's the way we like it in unix-land; if you can't install anything to your OS, you can't screw up your OS. But assuming everything went well in step 5, step 6 starts with you logging in as an administrator. You do that in most unix-like OSes like this:

su -

(and in case you're reading these instructions from windows, it's

runas /user:Administrator bash

)

Either way, you'll have to enter your administrator's password. You then do this:

cd /home/you/clfs/build/git-1.7.8.2
make install

And if all goes well, you now have git. Congratulations! To test, run

git --version

 

To review, the six steps were:

  1. Download all the sources
  2. Check the dependencies
  3. Copy the instructions into a spec
  4. Extract the source into the build directory
  5. Build using the spec
  6. Installation

 

Version Control and Backups

Now that we have Git, "the stupid version control system" (according to Linus Torvalds, Git's author), we can go ahead and put it to good use. Let's first initialize our repository. We can do this one of two ways: big repo, or small repo. By the way, we're done talking about building, so if you haven't already done so and you still need it, now would be a good place to go back and build curl.

Big repo:

If you're coming from my instructions above, don't use the big repo, skip down to the instructions for the small repository. You've only got four gigs to work with. You'll want roughly 12 gigs available for this. That's the obvious disadvantage; your repo will be huge. The advantage is that the "time machine" controls are going to be much simpler.

First, let's set up our repo. You need write permissions to your $CLFS directory.:

cd $CLFS
git init

That's it. Every time you complete a book chapter from here on out, you'll do this:

cd $CLFS
git add -A
git commit -m "completed 5.9, mpfr"

Every time you want to rewind things to the last time you committed, it's

cd $CLFS
git reset --hard

And every time you want to go back in time further than that, you'll do

cd $CLFS
git log | less

You'll see a long string of hexadecimal numbers, and then the log message you left yourself earlier. Pick out the "date" you'd like to return to, copy the first eight or so numbers from that long hexadecimal string (we'll say it's 1234ABCD), and then

git reset --hard
git checkout 1234ABCD

The big repo and small repo instructrions aren't mutually exclusive, so read on if you'd like to set up another repo for your specs and sources.

Small repo:

Let's initialize,

cd ~/clfs
git init

We're going to want to ignore several temporary files that get created during the build process:

cat >> .git/info/exclude << "DONE"
clfs/book/
clfs/build/
DONE

The crash course at the end of the Big Repo section should get you through here as well. If you'd like a gentle introduction to git that's a little more in depth than what I've given you, here's one:

http://thinkvitamin.com/code/starting-with-git-cheat-sheet/

One thing you will want to do differently, if you're using a small repo and not a big repo, is back up your $CLFS fileystem. If you have 4 gigs of free space elsewhere on your disk, and you are building into a 4 gig filesystem, a great way to back things up is with the dd tool. Please read the next 2 paragraphs before running this command, and modify the command to suit your needs. This one's safe, since you're reading from something in your device folder, not writing to it, if your hackles were raised (and good for you if they were):

dd if=/dev/sdb1 of=backup.ext2 bs=32M

The first time you do that, you'll see the problem with it: it takes forever and hogs system resources. I mentioned it only because it works, and it's already available to you. So instead, I'd like to suggest installing yet another program, rsync. If you don't already have it, its cblfs book page is here:

http://cblfs.cross-lfs.org/index.php/Rsync

and it has no dependencies. Rsync only copies over what's new and what's changed and skips everything else, so backing up is very fast. You can use it to create back-ups like this (assuming you're backing up to ~/clfs_backup) :

rsync -r --progress -v $CLFS ~/clfs_backup

As a bonus, you'll notice it gives you a progress bar and lets you know what's copied already. Once everything's in the clfs_backup directory, you can either set up a big repo there, or you can create tarballs of that directory, or you can burn it to cd. The power is yours and you have dozens of options.

 

RPM