AUTHOR: David Rosal i Ricart DATE: 2005-10-24 LICENSE: GNU Free Documentation License Version 1.2 SYNOPSIS: Using paco in ALFS PRIMARY URI: http://paco.sourceforge.net/doc/ DESCRIPTION: The program paco can be used along with nALFS to track the installation of an entire LFS system. This is done by editing the XML ALFS profiles properly. This hint intends to provide instructions to edit those profiles and integrate paco in ALFS in the most neat and safe way. ATTACHMENTS: * The LFS-6.1 profile paco patch: --> http://paco.sourceforge.net/dl/profile-LFS-6.1-paco.patch PREREQUISITES: * The program nALFS >= 1.2.4 --> http://www.linuxfromscratch.org/alfs/downloads/stable/ * The ALFS profiles for LFS-6.1: --> http://www.linuxfromscratch.org/alfs/downloads/profiles/stable/ * The program paco. Version 1.10.0 or later. --> http://sourceforge.net/project/showfiles.php?group_id=118115 HINT: 0. INTRODUCTION =============== There are mainly two ways to integrate paco in ALFS: the "brute force" way, and the neat way. In this hint I'm going to describe the neat way in detail, but first I wanted to say a few words about the "brute force" way, which indeed works fine, and has been happily used by some ALFS'ers until today. Note: Regardless the method used, before running nALFS, the tarball of paco has to be copied into the directory where the other LFS packages reside. 1. THE BRUTE FORCE WAY ====================== In a nutshell, this method consists of (ab)using the element to wrap each command that installs files into the system. As an example, let's look at the following piece of XML "code", taken from the file chapter06/binutils.xml in the LFS-6.1 profile, which installs the package binutils: ---[ begin XML ]--- tooldir=/usr install ../&binutils-directory;/include/libiberty.h /usr/include ---[ end XML ]--- The above fragment could be paco'ized thusly: ---[ begin XML ]--- -lp binutils-&binutils-version; make tooldir=/usr install -lp+ binutils-&binutils-version; cp ../&binutils-directory;/include/libiberty.h /usr/include ---[ end XML ]--- If you're familiar with paco, this is quite straightforward. With this method is always possible to track package installations, no matter the complexity or number of the commands involved, but it's not very... you know, clever, because it does not take advantage of the variety of elements defined in the ALFS DTD, and replaces each of them with the element . Next I'm gonna describe in detail the process of building an LFS system with nALFS and paco in a definitely better way. 2. THE NEAT WAY =============== [ Note for impatients: Go directly to section 2.1 of this hint. ] 2.0. Technichal notes ===================== To log package installations, paco uses a shared library to wrap the commands that install files into the system. Internally, right before spawning those commands, paco sets the following environment variables: LD_PRELOAD contains the path to the mentioned libpaco-log shared library, which is responsible to catch the system calls that install files into the system. Once libpaco-log is preloaded, it retrieves the rest of the variables from the environment. PACO_TMPFILE contains the path to the intermediate temporary log file used to comunicate libpaco-log and paco. If PACO_DEBUG is set to "yes", then libpaco-log prints debugging information. PACO_INCLUDE and PACO_EXCLUDE are a colon-separated list of paths to include or exclude when logging the created files. What we'll going to do in ALFS is to use the XML element to set those variables "by hand". From now on, the files in the LFS-6.1 profile will be referenced using paths relative to the LFS-6.1 profile directory, as created when unpacking the profile-LFS-6.1.tar.bz2 tarball. All of PACO_* variables may be set in the file LFS.xml, at the beggining of each chapter in which any package is installed, though it does not hurt if they are set throughout the whole LFS installation. The variable LD_PRELOAD must be set separately in each XML profile in which any package is installed. More strictly, it must be set whenever any install command (or group of them) are executed whithin those profiles. To ensure that the libpaco-log library is loaded only during the execution of the install commands, we have to join them toghether in a separate . Thus, the LD_PRELOAD variable is only set whithin that stage, and we prevent the log to be contaminated with all the files created during the configure, patch and build commands. Currently, in the LFS-6.1 profiles, the whole process of building and installing a package take place in a single stage called "Installing". We're going to split that stage into different smaller pieces, defining these new kinds of stages: "Building": Patches are applied. The package is configured and built. "Installing": LD_PRELOAD is set, and the package is installed. "Logging": Paco is used to create a log for the installed package, reading the list of files to log from the PACO_TMPFILE. The stages "Unpacking" and "Cleanup" are not modified. The "Building" stage can be omitted in those packages that don't need to be configured or built, like lfs-bootscripts or man-pages. IMPORTANT: The file PACO_TMPFILE must be removed at the beggining of the "Installing" stage, and at the end of the "Logging" stage, to clear the list of files to log. See an example of how should it be done in section 2.3 below. 2.1. Patching the LFS profile ============================= All changes can be done automatically in one step, by applying the patch profile-LFS-6.1-paco.patch from the LFS-6.1 profiles directory: $ patch -Np1 -i This hint could finish here, since the above patch makes all needed modifications, but I'll give detailed information on what is actually done by that patch, for those who want to know more, or tune it to their own preferences. 2.2. Cookbook to make it by hand ================================ We'll consider that we are not interested on tracking the installations of the packages in chapter 5 ("Constructing a Temporary System"), so all XML files in directory chapter05 that install packages will be left untouched. chapter05/creatingtoolsdir.xml ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ An additional element has to be added, to copy the paco package into the LFS partition: ---[ begin file fragment ]--- &ncurses-package; &paco-package; &patch-package; ---[ end file fragment ]--- config/package.ent ~~~~~~~~~~~~~~~~~~ The following 3 lines have to be added: ---[ begin file fragment ]--- ---[ end file fragment ]--- config/paco.ent ~~~~~~~~~~~~~~~ We have to create the file config/paco.ent, containing the paco related entities: ---[ begin file ]--- &paco-tmpfile;"> &paco-lib; "> -lD < &paco-tmpfile; &paco-clear;"> ---[ end file ]--- chapter06/paco.xml ~~~~~~~~~~~~~~~~~~ Paco itself should be installed as the first package in chapter 6, so we have to create the file chapter06/paco.xml, with the commands needed to install it: ---[ begin file ]--- %general_entities; %package_entities; %paco_entities; ]> &packages_dir;/&paco-package; &build_dir; &build_dir;/&paco-directory; --prefix=/usr --disable-gpaco --sysconfdir=/etc install /etc/pacorc =LOGDIR=&paco-logdir; =INCLUDE=&paco-include; =EXCLUDE=&paco-exclude; &build_dir;/&paco-directory; ---[ end file ]--- LFS.xml ~~~~~~~ The file LFS.xml has to be edited to include the installation of paco, right after the 'basic-dev' section in chapter 6: ---[ begin file fragment ]--- ---[ end file fragment ]--- The paco environment variables have to be set in chapters 6, 7, 8 and 9. For each of these chapters, in the element, the following variable settings must be inserted: ---[ begin file fragment ]--- &paco-debug; &paco-tmpfile; &paco-include; &paco-exclude; ---[ end file fragment ]--- chapter0?/*.xml ~~~~~~~~~~~~~~~ Finally, the hard work: Each and every XML profile in which any package is installed has to be modified, if we want that package to be logged. Since not all packages are installed the same way, it's not easy to make the modifications in an automatized way. An example of how should it be done in practice follows. 2.3. Example ============ Let's take the file chapter06/zlib.xml. I have stripped some parts like the "Patching" stage and the md5 checksum to make it simpler. This file is a good example, because install and non-install commands take place alternatively. Whithout using paco the file looks thusly: ---[ begin file ]--- %general_entities; %package_entities; ]> &packages_dir;/&zlib-package; &build_dir; &build_dir;/&zlib-directory; --prefix=/usr --shared install /lib/libz.so ../../lib/libz.so.&zlib-version; /usr/lib/libz.so clean --prefix=/usr install /usr/lib/libz.a &build_dir;/&zlib-directory; ---[ end file ]--- And here's the same file, but using paco to track the installation. Newly added lines are marked with '+' at the beggining. Modified lines are marked with '!': ---[ begin file ]--- + + %paco_entities; %general_entities; %package_entities; ]> &packages_dir;/&zlib-package; &build_dir; ! &build_dir;/&zlib-directory; --prefix=/usr --shared + + + &paco-ld-preload; &build_dir;/&zlib-directory; + &paco-clear; install /lib/libz.so ../../lib/libz.so.&zlib-version; /usr/lib/libz.so + + + + &build_dir;/&zlib-directory; + clean --prefix=/usr + + + + &paco-ld-preload; + &build_dir;/&zlib-directory; + install /usr/lib/libz.a + + + + &build_dir;/&zlib-directory; + + &paco-log-dirname; &build_dir;/&zlib-directory; ---[ end file ]--- Let's analyze the different changes that have been made: First, the paco entities are included. This is done by the lines: %paco_entities; The original "Installing" stage, has been splitted into different stages: # "Building" (1st pass): The package is configure'd and make'd. No files are installed into the system, so we leave the commands untouched. # "Installing" (1st pass): Some files are installed, so we have to set the LD_PRELOAD environment variable at the beggining of the stage. This is done by the XML "macro" paco-ld-preload: &paco-ld-preload; &build_dir;/&zlib-directory; And because it's the first "Installing" stage in this file, the libpaco-log temporary file is removed to ensure that we don't log any file installed by any previously installed package: &paco-clear; # "Building" (2nd pass): The package is reconfigured again in order to install the static library. The LD_PRELOAD variable is automatically unset because the environment set in the of the previous stage is not seen by the current one. # "Installing" (2nd pass): The static library is installed. We set the LD_PRELOAD variable again, but this time "&paco-clear" is not used, because we do not want to overwrite the already logged files, but to add new files into the log. # "Logging": This is a completely new stage. Now that the list of files to be logged is saved in the libpaco-log temporary file, a "paco -lp" command creates the log for the package. This is done with the XML "macro" paco-log-dirname. It's worth noting that this macro works when the name of the directory from which the package is built is the same as the name of the package as we want it to be logged. This is generally true for all packages, with the following exceptions: - vim and tcl: because the directory created when unpacking the tarballs has a "non standard" name (vim63 and tcl5.8.3). - All packages that need to be built in a dedicated directory, separated from the sources (gcc, glibc, e2fsprogs). In these cases, the macro &paco-log-dirname; should be replaced by the following lines: -lp foo-&foo-version; < &paco-tmpfile; &paco-clear; where 'foo' is the name of the package to log. 3. THE END ========== This hint finishes here. Now it should be up to the reader to build an entire LFS system with a single press on the Enter key, having control over almost every byte written in the hard disk. Ha! And this is only the beggining. With litle modifications, the above instructions can be applied to install all the BLFS stuff too... ACKNOWLEDGEMENTS: I'd like to thank specially the ALFS staff. I'm really impressed with the possibilities of the ALFS system. No more nights wasted in front of the computer screen, typing "./configure && make && make install" like a zombie... I'd also like to thank Brad Bailey for creating an initial version of the alfs-paco patch, and to encourage me to write this hint. And finally, I can't forget the people who aid me in the development of paco. CHANGELOG: [2005-10-24] * Initial hint.