Windows Environment Set-Up [June 1, 2012, 2:31 p.m.]

Cross linux from scratch can be built from just about any operating system with a working C compiler. I'll walk you through a Windows, MinGW/MSYS environment setup. It's probably easiest if you're having trouble, though, to skip this task and do this from a BSD, Mac, or a Linux computer. Linux has several LiveCD, LiveUSB, and virtualized distributions that are worth checking out. I think Linux from Scratch even has their own LiveCD made just for us. Any of these will have working c compilers and environments that will suit our needs.

I made a quick video in three parts weighing in at about half an hour helping you if you're starting out the Windows way. You might want to full screen. If you're not doing this on windows, you can skip down to the next section.

http://youtu.be/U2Agy6ulJtc

One thing I neglected to mention (since I didn't realize I needed it until later), you'll also want to get the "xz" and "mingw-developer-toolkit" packages with mingw-get, in an admin shell, like this:

mingw-get install xz
mingw-get install mingw-developer-toolkit

http://youtu.be/it4kGicvADI

I didn't get around to it in the next video, but you can chain configure and make together with a double-ampersand like

./configure --all-your-config-options && make

http://youtu.be/-Tl2WsGfw1M

Ext2 File System

We'll be building CLFS on an ext2 filesystem. Windows doesn't have to be able to access ext2 directly, but I'll show you how if you'd like to be able to. You will need to be able to create and populate ext2 filesystems, though, and I'll show you how to do that as well.

A filesystem is a way for an operating system to store files, folders, and information about those files and folders such as security permissions and when a file was last accessed. Windows has two native filesystems, FAT and NTFS. Linux can work with both of these, but ext2 is still preferable. The advantage to using ext2 is that just about any disk repair and boot loader utility can read it. This means that if anything ever goes wrong with your computer, you can always download a LiveCD and fix any problem with your disk itself (except maybe fire and explosion ;) ). It's also easier to use Linux's native disk maintenance utilities with ext2.

If you want to be able to use ext2 drives or partitions natively from windows, there's two options available to you. I've had more luck with Ext2FsD than Ext2IFS on Windows 7 64 bit personally. If neither of these work for you you can still build CLFS on an ext2 filesystem from windows, you just won't be able to access the OS you build directly from windows. Remember to install Ext2IFS in compatibility mode for Windows Vista Service Pack 2 if you're on windows 7.

http://www.fs-driver.org/

http://www.ext2fsd.com/

Even if neither of those are installed, you can create and populate an ext2 filesystem from windows using Cygwin. You can grab cygwin from here:

http://cygwin.com/install.html

Cygwin installation is a little involved for this tutorial (which is why I linked to the install help page instead of the download page), but make sure that these two things are true:

You've installed cygwin to a different top-level directory from mingw/msys (in other words, something like C:\cygwin and C:\mingw would work)
You've installed the e2fsprogs and e2fsimage packages, found under "System"

Here's a short video tutorial that will guide you through everything once you have all that set up. Please note that, although you should be safe if you follow the instructions in the video, it contains a potentially unsafe command at the end. The potentially unsafe command is optional, but you can't make a bootable OS without it. If you follow the instructions in the video you should be fine, the risk is minimal. However, this would be great place for me to recommend you back up any important files and have a recovery disk for your OS handy.

Installing

Whatever you're installing LFS to, it would help a lot if it were bootable. I picked an 8 GB USB stick. It doesn't have to be formatted, and in fact it would help a lot if you didn't care what was on it.

I will not be teaching you how to create LiveCDs, although there are tutorials that exist online, including here:

http://www.linux-live.org/

You can of course ignore all tasks past program installation. This will give you a "chroot" distribution, or a distribution you can use as an embedded distro from another linux distro with a compatible kernel via the "chroot" command. It's up to you if you want a bootable OS.

That's it; everything else we'll be building for ourselves.

Hole Hawg

Feel free to skip this section; it's just "flavor text".

I wasn't trying to scare you off in the video in that last section. One thing about the GNU tools is that they are VERY powerful, but "with great power comes great responsibility" as Spider Man's Uncle Ben would say. dd happens to be one of the most powerful tools GNU has to offer, and as a result, it's sometimes jokingly called "destroy disk". You may be asking yourself why anyone would use these tools. The answer is that, while it's true the "safety is off" so to speak, it's also true that when you are using an open source OS, it is VERY easy for you to recover data, fix your own computer problems, and in general recover from any terrible thing you've done. And unlike with a windows system, if something goes wrong, it's almost always because you did something wrong.

The analogy I like to use here is a kitchen. As a windows user, you've been buying frozen food. I'm showing you how to use the oven. If you aren't careful, you might set your kitchen on fire. But the advantages that come with being able to cook your own food and save money far outweigh the disadvantages. And if you're really determined, you can start a fire with your microwave anyway ;)

One thing that I think is a must-read for anyone getting their feet wet with Unix is Neal Stephenson's "hole hawg" portion of his essay, "In the beginning there was the command line". I've linked to it here:

http://www.cryptonomicon.com/command.zip

Unzip it and start on line 464, where it says "THE HOLE HAWG OF OPERATING SYSTEMS" in bold letters.

That's pretty much all I wanted to say here. You're building a Genie. Standard Genie rules apply.

Goal

In this stage, we lay out our foundation. Our goals are all of the following:

A "bootstrapping" filesystem that closely mirrors the one found in the book, makes it easy to keep track of what we're doing, and allows us to build programs as an ordinary user while making sure the only user that can install software is an administrator. This usually has the creative name "build environment".
A way to "rewind" in case we mess anything up. This is usually done with something called "version control".
A way to modularize components so that they can be used on multiple systems. This is called "packaging".

We'll be using git for version control, and RPM for packaging.

Build Environment

Okay, whether you're here from a Windows, BSD, Mac, Linux, or other environment, the rest of the commands are going to be the same. If you haven't run the version-check.sh script from a bash shell successfully yet, you don't belong here; please skip back to previous parts of the tutorial. The videos I made for MinGW are Windows-centric, but contain useful hints from other OSes as well.

To start off with, we're going to make a folder with the following directory tree layout.

You can do this from your home directory with the following command:

mkdir -p clfs/{book,build,patches,scripts,sources,specs}

If that last command didn't work for you, it's possible you're running from a busybox bash shell. I'll be using {,} substitution frequently through these instructions, and it would be easiest to follow along if you went ahead and installed a proper bash shell. You can check to see which bash shell you have with the command

bash --version

Next, if you haven't already done so, go ahead and populate the sources and patches directories like so:

cd clfs/sources
wget http://greenback.gremlin.net/downloads.txt
wget -i downloads.txt
mv *.patch ../patches

Next, if you haven't already done so, go ahead and populate the scripts directory like so:

cd
cd clfs/scripts
wget http://greenback.gremlin.net/chown
wget http://greenback.gremlin.net/RENAME.profile
wget http://greenback.gremlin.net/version-check.sh

Finally, we'll add the seed spec to the specs folder:

cd ~/clfs/specs
wget http://greenback.gremlin.net/seed.spec

It might be a good idea right now to back up your clfs directory (everything we just created). How this is done is up to you. Two good options are creating a tarball, and burning everything to CD or DVD.

Next, we're going to need to set up our clfs directory. First, let's go ahead and create it in a shell with admin rights:

mkdir -p /mnt/clfs

If you are actually making CLFS on another device, you can go ahead and mount it now. This is going to be environment dependent. In an MSYS environment, as an administrator, assuming I wanted to build on my USB stick which Windows recognizes as my H drive, it's

echo "H:/ /mnt/clfs" >> /etc/fstab

In most unix-likes, assuming I have a USB stick which the OS recognizes as /dev/sdb1, as an administrator (or for you hotshots, an ordinary user with permissions set up correctly in /etc/fstab) it's

mount /dev/sdb1 /mnt/clfs

If this is your first build-through, or either of the above options applies to you, go ahead and skip the next two paragraphs, to the sentence that starts "Once you've got your clfs directory ...".

If you're building CLFS in another directory, you can either remove your clfs directory and make /mnt/clfs a symbolic link, or you can loop mount. The sytnax for loop-mounting in MSYS is exactly the same as mounting a device. Just make sure no paths have spaces and use short path names (like "Progra~1") if there are spaces. In a unix-like, as an admin and assuming I'm really building in a folder called clfs in my home directory, it's

mount --bind ~/clfs /mnt/clfs

Once you've got your clfs directory made and mounted, you're going to need to make some directories. For this command, you can use either an ordinary user or an admin shell. Just be sure you've got permission to write to the directory /mnt/clfs.

mkdir -p /mnt/clfs/{bin,usr/bin,tools,cross-tools}

Next, we'll need to create a couple links. Msys users, don't use ln -s. In the second video above, I tell you how to use mklink to create symbolic links. Otherwise, you can always create a mount point. So from msys, as an admin, you'll either do:

cd /
cmd
mklink /J tools H:\tools
mklink /J cross-tools H:\tools

mkdir /{,cross-}tools
echo "H:/tools /tools" >> /etc/fstab
echo "H:/cross-tools /cross-tools" >> /etc/fstab

If you are NOT using MSYS, it's a little easier (as root):

ln -sv /mnt/clfs/tools /tools
ln -sv /mnt/clfs/cross-tools /cross-tools

The book recommends creating a clfs user at this point. If you're interested in doing this, it's in section 4.4. This is entirely optional and won't affect any of my instructions.

Section 4.5 has in depth advice for setting up your environment. For my instructions, I will assume you only have one profile script. If you are not in an msys environment, I'd like to advise you to set up a bashrc script so that it simply sources your .bash_profile script, like so:

cat > ~/.bashrc << "EIO"
#!/bin/bash

. ~/.bash_profile
EIO

And as I mentioned earlier, in msys the name of the script is ~/.profile . For me, you only need to make sure that inside your profile script (either ~/.profile or ~/.bash_profile), your PATH environment variable leads off with "/cross-tools/bin" and that you have your CLFS environment variable set. The two lines might look like this (not literally; the "..." expands into my full PATH variable; I just wanted to get across that /cross-tools/bin is at the front):

export PATH="/cross-tools/bin:.:/usr/local/bin:/mingw/bin:..."
export CLFS=/mnt/clfs

You also need the following line in there somewhere:

set +h

For msys, I have an example profile script above. It's called RENAME.profile2. For other operating systems, you can examine this script and the three lines I just mentioned.

Once you have your profile script written, you source it like so (assuming a linux environment here just to be different, msys users substitute .profile):

. ~/.bash_profile

Okay! We have our environment completely set up and we're almost done with this task. First, though, before we're completely ready to build CLFS, let's make a couple tools from source that will help us out a lot.

Git and the Generic 6 Step Package Build Process

While you will want a git binary available to you, compiling from source in this section is entirely optional. Depending on how "from scratch" you'd like to be, there are precompiled or pre-packaged versions of git available to just about every operating system. Git hosts downloads here,

http://git-scm.com/downloads

Windows users only

First, when you install the binary you get from the git website, you'll want to make sure that the "Run git from the Windows Command Prompt" radio button is selected, followed by the "Checkout as-is, commit Unix-style line endings" button on the next screen.

Second, this time, it's you MinGW/MSYS that catch a break and get to skip ahead to the next section. You are welcome to try to build git on windows, but I was unsuccessful under MinGW/MSYS. I'm going to use the pre-compiled binary. If you want to try, one thing I had to do was create a config.cache file with the following contents:

git_cv_socklen_t_equiv=int

and then add the configure option,

--cache-file=config.cache

However, my 64 bit windows would still not include winsock2.h correctly. You're also going to need zlib, which you can install using mingw-get.

Otherwise, there is a great development community specifically for porting git to windows, including building it from source, here

http://code.google.com/p/msysgit/

and if you're comfortable trying your luck with cygwin instead of msys (bearing in mind that your git can only be used in your cygwin environment and not your msys environment), there's a tutorial I haven't tried here,

http://techblogging.wordpress.com/2008/04/11/compiling-and-installing-git-on-windows-under-cygwin/

What you can do is read through this section for generic package building instructions, just realize that it's not going to apply directly to you. The git download page I linked earlier has a link to the windows binary, and I've provided a direct link here as well:

https://github.com/downloads/schacon/testy/Git-1.7.10-preview20120409.exe

Everyone else

So if you're still reading this section, you're in a linux, BSD, or mac environment, or you're just here to learn.

There's more flavor text in the next paragraph, feel free to skip it.

One of the biggest complaints people have building LFS is that it's frustrating when a package doesn't build correctly, and you have to scrap everything and start over. Some people may have joked that it would be a lot easier with a time machine. Well guess what! Git gives us a kind of a "time machine". You can undo and redo operations, create multiple timelines, merge them back together, cherry pick and ressurect anything you've done in the past, and resolve conflicting visions between you and your fellow time travelers, among other things. It has a little bit of a learning curve to use it to its full potential, but we'll only be using very simple commands.

Feel free to just install git the easy way and skip to the next section if you have any problems here. My rational for building for source here, is to A.) ensure that our build environment works, and b.) give you a gentle walk-through of the unix packaging process. We're going to create what's called a "source controlled package". This section and the troubleshooting section will be a reference tool for you later if you have any problems.

We're going to use the page from the "CBLFS" book to build this package. The "B" stands for "beyond". It's appropriate, then, that we're pulling the instructions for creating our time machine from the future, in the book intended to be used once the Cross Linux From Scratch system has already been built.

Enough talk. Allons-Y!

The CBLFS page for git is here:

http://cblfs.cross-lfs.org/index.php/GIT

By the way, that's also an easy way to look up generic build instructions for any linux package ever; just type in

http://cblfs.cross-lfs.org/index.php/<name_of_package>

The first letter is usually capitalized, and spaces are underscores. If you don't find the package that way (I didn't in this case because GIT is all capitals), just use the search box in the lower left hand corner. If you come across a rare package that isn't up there already, and you figure out how to build and install it, consider making a CBLFS page yourself.

Step 1: Grab Sources

According to the CBLFS book page, the sources are located at

http://git-core.googlecode.com/files/git-1.7.8.2.tar.gz

http://git-core.googlecode.com/files/git-manpages-1.7.8.2.tar.gz

We'll go ahead and grab those.

If it feels like I'm talking a little fast at this point, this would be a good place to review the links in the "prerequisites" section. Things are going to get a little technical from here on out.

First, fire up a shell as an ordinary user. We'll want to be an ordinary user for steps 1-5. Then:

cd ~/clfs/sources
wget http://git-core.googlecode.com/files/git-1.7.8.2.tar.gz
wget http://git-core.googlecode.com/files/git-manpages-1.7.8.2.tar.gz

That's going to be step 1 whenever we grab anything from the CBLFS book, by the way; grab all the sources listed at the top of the page, and dump them in our clfs/sources folder.

Step 2: Check Dependencies

Step 2 is going to be looking at the dependencies section. In this case, there's 2 required packages:

Curl
Expat

You'll want to use your OS's standard package management tools to get those. If you're reading this and you've been following instructions, you've already got expat. If you want to try your luck building Curl from scratch, here's its CBLFS page:

http://cblfs.cross-lfs.org/index.php/Curl

If you're doing things the hard way (go you!), the build instructions for Curl are going to be nearly identical to the ones for git, so just keep reading. if you need help building packages from source, you can use the second video in the Environment section for help. There, I build the ncurses package from source on windows, but the same general procedure should work for you too.

Step 3: Write a spec

Once you've got all your dependencies installed, step 3 is to copy the build instructions into a packager script. Since we don't have RPM installed yet, we're going to make a shell script for this. I'm doing things this way in case you decide you'd rather not use RPM, so that you'll have some way of packaging. You can always use shell scripts for packaging no matter what.

For this tutorial I'll be building a non-multilib system. As a challenge to yourself, if your processor can handle 64 bit libraries, you can go back and build a multilib system later. Let's go to the git page, highlight all the instructions under "non multilib", and copy them. Then we'll do this in our shell:

cd ~/clfs/specs
cat > git.sh << "EIEIO"
Compile the package:
./configure --prefix=/usr --libexecdir=/usr/lib \
    --sysconfdir=/etc &&
make
Install the package
make install
Install the man pages:
tar xvf ../git-manpages-1.7.8.2.tar.gz -C /usr/share/man
EIEIO

Note that you don't have to type in all of that; everything between the EIEIOs I just pasted in.

Next, let's open that file we just made (it's called "git.sh") in our favorite text editor. My personal favorite is vim. Other good command line choices are emacs, nano, and joe. If you open in a graphical editor, make sure there's no text formatting going on. If you've got KDE, I like kate.

#!/bin/bash

set -e

# This should be /usr for bsd, mac, or linux, or /mingw on windows
PREFIX=/usr

#Compile the package:
./configure --prefix=$PREFIX --libexecdir=$PREFIX/lib \
--sysconfdir=/etc && make

# Install the package
make install

# Install the man pages:
tar xvf ../git-manpages-1.7.8.2.tar.gz -C /usr/share/man

What we did there is:

added the line "#!/bin/bash" at the beginning
- The "#!/bin/bash" line is called the "shebang". It says that when we run the script as an executable, we should use /bin/bash to interpret it. So if this were a python script instead, we'd have something like "#!/usr/bin/python".
added the line "set -e"
- "set -e" causes the script to exit immediately if something goes wrong.
added a few "#" marks at the beginnings of a few lines.
- These are what are known as "commenting out a line". The bash interpreter, /bin/bash, ignores everything after one of those hash marks. If you look at the git CBLFS page, you'll notice I put one of those marks everywhere the line was a comment not meant to be run, so that only the real commands get executed. I like leaving those in there, because they keep the script real easy to read if I have to make changes later.
Set up a PREFIX environment variable, and changed --prefix=/usr to --prefix=$PREFIX
- This is the directory you're installing to. The entire directory tree for git is going to be under this prefix. So binaries will go into $PREFIX/bin, c and c++ headers will go into $PREFIX/include, and libraries will go into $PREFIX/lib. Sometimes packages need other configuration files. Those go into folders called etc, var, and shared. We'll go over those later on when we start build CLFS. I left some suggestions as to what PREFIX should be in the comment I added.
  - ADVANCED: One other suggestion, if you're trying to do everything as an ordinary user, is to change this directory to one you have write permissions on as an ordinary user. Just be sure you know what you're doing. This is a little advanced for this tutorial, but if you go this route, you'll need to set your PATH up so that your binaries are on it, and you'll need to setup your CFLAGS variable to include -I${PREFIX}/include and your LDFLAGS variable to include -L${PREFIX}/include. Those shouldn't be substitutions, by the way; go ahead and expand them because you'll be setting PREFIX to other things.

We're going to need to make one more quick change to that last line. If you recall, our git-manpages tarball is inside our sources directory. We also (probably) don't have permission to install anything as an ordinary user, so we'll just echo the command so that we can copy/paste it into an admin shell later. For quick changes, I like to use the sed tool instead of firing up my editor. We can edit everything like this:

sed -e 's@\.\./git-manpages@~/clfs/sources/git-manpages@' \
    -e 's/^tar.\+/echo "&"/' \
    -e 's/make install/echo "&"/' \
    -i git.sh

The sed tool relies on something called "regular expressions".
You can look those up online if you'd like to learn how to use the sed tool for yourself. 

Step 4: Extract the package's build directory

Okay. Now that we've prepared our installer script, which I'll call "spec" from now on, it's on to step 4. Step 4 is changing to our build directory and extracting.

cd ~/clfs/build
tar xvf ../sources/git-1.7.8.2.tar.gz
cd git-1.7.8.2

Easy, right?

Step 5: Build the package

Step 5 is the one most likely to give you headaches. So if anything goes wrong, the entire troubleshooting section below is dedicated just to step 5 of the build process here. I've bolded out the stuff that's most likely to go wrong to make it easier to find.

All that buildup, you ready for step 5?

sh ../../specs/git.sh

Step 6: Installation

Step 6, is to install everything we just built. If you were following instructions earlier, you're an ordinary user and don't have the authority to install anything. That's the way we like it in unix-land; if you can't install anything to your OS, you can't screw up your OS. But assuming everything went well in step 5, step 6 starts with you logging in as an administrator. You do that in most unix-like OSes like this:

su -

(and in case you're reading these instructions from windows, it's

runas /user:Administrator bash

)

Either way, you'll have to enter your administrator's password. You then do this:

cd /home/you/clfs/build/git-1.7.8.2
make install

And if all goes well, you now have git. Congratulations! To test, run

git --version

To review, the six steps were:

Download all the sources
Check the dependencies
Copy the instructions into a spec
Extract the source into the build directory
Build using the spec
Installation

Version Control and Backups

Now that we have Git, "the stupid version control system" (according to Linus Torvalds, Git's author), we can go ahead and put it to good use. Let's first initialize our repository. We can do this one of two ways: big repo, or small repo. By the way, we're done talking about building, so if you haven't already done so and you still need it, now would be a good place to go back and build curl.

Big repo:

If you're coming from my instructions above, don't use the big repo, skip down to the instructions for the small repository. You've only got four gigs to work with. You'll want roughly 12 gigs available for this. That's the obvious disadvantage; your repo will be huge. The advantage is that the "time machine" controls are going to be much simpler.

First, let's set up our repo. You need write permissions to your $CLFS directory.:

cd $CLFS
git init

That's it. Every time you complete a book chapter from here on out, you'll do this:

cd $CLFS
git add -A
git commit -m "completed 5.9, mpfr"

Every time you want to rewind things to the last time you committed, it's

cd $CLFS
git reset --hard

And every time you want to go back in time further than that, you'll do

cd $CLFS
git log | less

You'll see a long string of hexadecimal numbers, and then the log message you left yourself earlier. Pick out the "date" you'd like to return to, copy the first eight or so numbers from that long hexadecimal string (we'll say it's 1234ABCD), and then

git reset --hard
git checkout 1234ABCD

The big repo and small repo instructrions aren't mutually exclusive, so read on if you'd like to set up another repo for your specs and sources.

Small repo:

Let's initialize,

cd ~/clfs
git init

We're going to want to ignore several temporary files that get created during the build process:

cat >> .git/info/exclude << "DONE"
clfs/book/
clfs/build/
DONE

The crash course at the end of the Big Repo section should get you through here as well. If you'd like a gentle introduction to git that's a little more in depth than what I've given you, here's one:

http://thinkvitamin.com/code/starting-with-git-cheat-sheet/

One thing you will want to do differently, if you're using a small repo and not a big repo, is back up your $CLFS fileystem. If you have 4 gigs of free space elsewhere on your disk, and you are building into a 4 gig filesystem, a great way to back things up is with the dd tool. Please read the next 2 paragraphs before running this command, and modify the command to suit your needs. This one's safe, since you're reading from something in your device folder, not writing to it, if your hackles were raised (and good for you if they were):

dd if=/dev/sdb1 of=backup.ext2 bs=32M

The first time you do that, you'll see the problem with it: it takes forever and hogs system resources. I mentioned it only because it works, and it's already available to you. So instead, I'd like to suggest installing yet another program, rsync. If you don't already have it, its cblfs book page is here:

http://cblfs.cross-lfs.org/index.php/Rsync

and it has no dependencies. Rsync only copies over what's new and what's changed and skips everything else, so backing up is very fast. You can use it to create back-ups like this (assuming you're backing up to ~/clfs_backup) :

rsync -r --progress -v $CLFS ~/clfs_backup

As a bonus, you'll notice it gives you a progress bar and lets you know what's copied already. Once everything's in the clfs_backup directory, you can either set up a big repo there, or you can create tarballs of that directory, or you can burn it to cd. The power is yours and you have dozens of options.