The URL Changed: new URL is http://www.sysresccd.org/Howto-advanced-customization

1 Rescue CD Setup (Version 2)

This document describes how to customise and create a rescue CD. The rescue CD is based on http://www.sysresccd.org/ with a modified kernel to allow serial output to /dev/ttyS*. If you do not require a custom kernel, then the customization procedure as described in the System Rescue CDRom Manual will more than likely be adequate for your needs.

I have also implemented the grsecurity patches to give our setup a bit more robustness against buffer overflow attacks in the case that we want to enable a network connection while working on the remote machine.

2 My systems setup

Computer A (Redhat 9, CD Writer, USB connection, minicom running at 115200 baud on ttyS0)

Computer B (Barebones box, no OS required, CD-Rom, serial port and USB)

Hard-Disk with USB enclosure (to allow easy hotswapping of disk from one machine to another without reboot). My disk is partitioned with three ext3 partitions, and the holding space for the customization process will be on the third disk (of course you have some fee choice here). I have found it easier to have at least 2Gig free disk space, although customization can be done with 600Meg, I allow for the possibility of taking snapshots of the image file.

Null modem cable connecting Machine A to B.

3 Quick overview of steps involved

Download the Rescue CD iso file to CD Burner machine and burn a copy of the CD.

Add any kernel patches and/or kernel customizations.

Recompile the kernel with serial console enabled in the kernel.

Recompile the cloop.o module for this new kernel. The stock cloop.o will not work due to kernel differences.

Using the original Bootable rescue CD, generate a snapshot image file.

Add kernel modules to snapshot.

Enable serial console support in the snapshot at the OS level.

Add new kernel to the boot partition, add modules to initrd (including new cloop.o)

Update the isolinux.cfg file to boot with serial console at the desired speed.

Regenerate a new ISO file.

Burn new CD and test.

3.1 Download the latest sysressccd

The latest sysresccd can be found at [ http://www.sysresccd.org/download.en.php]. You can also choose to download a beta version of the CD which may have some extra feature and kernel updates. The ISO should be checked against the published md5 checksum. Burn a copy of the new CD - I used cdrecord here.

3.2 Prepare the kernel source

To enable serial console in the kernel, it is necessary to have kernel serial console support enabled (Note, I have been in contact with Francois Dupoux, the author of Sysresccd, and he has added serial console support to the kernel in the 0.26 stream). In addition, We will add Grsecurity patches to our kernel.

Since I only have one system running Linux which has all the kernel developers packages installed, I shall recompile my kernel on this machine. Naturally enough, to recompile the kernel we need a number of things:

The kernel source
The kernel config file # Unless you want to configure you own kernel
Any kernel patches that we want to install.

The kernel sources from were found at http://www.kernel.org. At the time of writing, the current kernel version is linux-2.4.22, I am not particularly keen on trying a 2.6 kernel yet. Assuming that you have already downloaded the kernel source, the following steps are simply paraphrased and must be modified for your setup:

   
   bunzip /path/to/downloaded/linux_source/linux-2.4.22.tar.bz 
   mv /usr/src/linux-2.4.22 /usr/src/linux-2.4.22-temp  
   cd /usr/src/ 
   tar xf /path/to/downloaded/linux_source/linux-2.4.22.tar 
   cd linux-2.4.22 
   cp http://www.sysresccd.org/kernel/sysresccd-X.Y.Z/kernel-sysresccd-X.Y.Z.config .config

Note that Step 2 is only required if you already have a kernel source directory or that name. I would like to preserve the orioginal source.

3.2.1 Patch the kernel with grsecurity kernel patches (Optional)

The kernel patches can be found at [ http://www.grsecurity.net/download.php]. Make sure you download the appropriate patch set for the kernel you are going to compile. At the time of writing the patch set is: grsecurity-1.9.12-2.4.22.patch.

cd /usr/src 
patch -p0 < grsecurity-1.9.12-2.4.22.patch

Note, this updates the kernel source in linux-2.4.22 source tree.

3.2.2 Modify the .config file

We shall use the .config file downloaded from www.sysresccd.org, and manually add in the Grsecurity options to the .config file. It is also possible, to add these options in later when running make oldconfig, i.e. If these options weren't added by you, you will be prompted as to whether you want to add them or not. Also, I do not recommend compiling the serial code as a module for this kernel, loading the serial modules at boot time does not give us access to /dev/console, so the machine will hang on boot. So make sure the following are set in .config

> CONFIG_SERIAL=y 
> CONFIG_SERIAL_CONSOLE=y 
> CONFIG_SERIAL_EXTENDED=y 
> CONFIG_SERIAL_MANY_PORTS=y 
> CONFIG_SERIAL_SHARE_IRQ=y 
> # CONFIG_SERIAL_DETECT_IRQ is not set 
> CONFIG_SERIAL_MULTIPORT=y 
> CONFIG_HUB6=y

In addition, for Grsecurity, the Config setting that I have for my .config file are as follows:

# 
# Address Space Protection 
# 
CONFIG_GRKERNSEC_PAX_NOEXEC=y 
CONFIG_GRKERNSEC_PAX_PAGEEXEC=y 
CONFIG_GRKERNSEC_PAX_SEGMEXEC=y 
CONFIG_GRKERNSEC_PAX_EMUTRAMP=y 
CONFIG_GRKERNSEC_PAX_EMUSIGRT=y 
CONFIG_GRKERNSEC_PAX_MPROTECT=y 
CONFIG_GRKERNSEC_PAX_NOELFRELOCS=y 
CONFIG_GRKERNSEC_PAX_ASLR=y 
CONFIG_GRKERNSEC_PAX_RANDKSTACK=y 
CONFIG_GRKERNSEC_PAX_RANDUSTACK=y 
CONFIG_GRKERNSEC_PAX_RANDMMAP=y 
CONFIG_GRKERNSEC_PAX_RANDEXEC=y 
CONFIG_GRKERNSEC_KMEM=y 
CONFIG_GRKERNSEC_IO=y 
CONFIG_RTC=y 
CONFIG_GRKERNSEC_PROC_MEMMAP=y 
CONFIG_GRKERNSEC_HIDESYM=y 
 
# 
# ACL options 
# 
# CONFIG_GRKERNSEC_ACL_HIDEKERN is not set 
CONFIG_GRKERNSEC_ACL_MAXTRIES=3 
CONFIG_GRKERNSEC_ACL_TIMEOUT=30 
 
# 
# Filesystem Protections 
# 
# CONFIG_GRKERNSEC_PROC is not set 
CONFIG_GRKERNSEC_LINK=y 
CONFIG_GRKERNSEC_FIFO=y 
CONFIG_GRKERNSEC_CHROOT=y 
CONFIG_GRKERNSEC_CHROOT_MOUNT=y 
CONFIG_GRKERNSEC_CHROOT_DOUBLE=y 
CONFIG_GRKERNSEC_CHROOT_PIVOT=y 
CONFIG_GRKERNSEC_CHROOT_CHDIR=y 
CONFIG_GRKERNSEC_CHROOT_CHMOD=y 
CONFIG_GRKERNSEC_CHROOT_FCHDIR=y 
CONFIG_GRKERNSEC_CHROOT_MKNOD=y 
CONFIG_GRKERNSEC_CHROOT_SHMAT=y 
CONFIG_GRKERNSEC_CHROOT_UNIX=y 
CONFIG_GRKERNSEC_CHROOT_FINDTASK=y 
CONFIG_GRKERNSEC_CHROOT_NICE=y 
CONFIG_GRKERNSEC_CHROOT_SYSCTL=y 
CONFIG_GRKERNSEC_CHROOT_CAPS=y 
 
# 
# Kernel Auditing 
# 
CONFIG_GRKERNSEC_AUDIT_GROUP=y 
CONFIG_GRKERNSEC_AUDIT_GID=1007 
CONFIG_GRKERNSEC_EXECLOG=y 
CONFIG_GRKERNSEC_RESLOG=y 
CONFIG_GRKERNSEC_CHROOT_EXECLOG=y 
CONFIG_GRKERNSEC_AUDIT_CHDIR=y 
CONFIG_GRKERNSEC_AUDIT_MOUNT=y 
CONFIG_GRKERNSEC_AUDIT_IPC=y 
CONFIG_GRKERNSEC_SIGNAL=y 
CONFIG_GRKERNSEC_FORKFAIL=y 
CONFIG_GRKERNSEC_TIME=y 
 
# 
# Executable Protections 
# 
CONFIG_GRKERNSEC_EXECVE=y 
CONFIG_GRKERNSEC_DMESG=y 
CONFIG_GRKERNSEC_RANDPID=y 
CONFIG_GRKERNSEC_TPE=y 
CONFIG_GRKERNSEC_TPE_ALL=y 
CONFIG_GRKERNSEC_TPE_GID=1005 
 
# 
# Network Protections 
# 
CONFIG_GRKERNSEC_RANDNET=y 
CONFIG_GRKERNSEC_RANDISN=y 
CONFIG_GRKERNSEC_RANDID=y 
CONFIG_GRKERNSEC_RANDSRC=y 
CONFIG_GRKERNSEC_RANDRPC=y 
CONFIG_GRKERNSEC_SOCKET=y 
CONFIG_GRKERNSEC_SOCKET_ALL=y 
CONFIG_GRKERNSEC_SOCKET_ALL_GID=1004 
CONFIG_GRKERNSEC_SOCKET_CLIENT=y 
CONFIG_GRKERNSEC_SOCKET_CLIENT_GID=1003 
CONFIG_GRKERNSEC_SOCKET_SERVER=y 
CONFIG_GRKERNSEC_SOCKET_SERVER_GID=1002 
 
# 
# Sysctl support 
# 
CONFIG_GRKERNSEC_SYSCTL=y 
 
# 
# Logging options 
# 
CONFIG_GRKERNSEC_FLOODTIME=10 
CONFIG_GRKERNSEC_FLOODBURST=4

3.2.3 Build the kernel

The build procedure for the kernel is described in the kernel source README file. The only caveats are:

If you are already running a 2.4.22 kernel, make sure that you rename the /usr/src/linux-2.4.22 directory (temporarily), otherwise you will overwrite your running kernel sources.

sh# mv /usr/src/linux-2.4.22 /usr/src/linux-2.4.22-resc 
sh# cd /usr/src/linux-2.4.22-resc;

At this stage you must edit the top level Makefile, and change the parameter EXTRAVERSION on line 4 to read

EXTRAVERSION = -resc

If you do not make this change you run the risk over writing any installed 2.4.22 kernel during the install and modules_install steps

mkdir /lib/modules/2.4.22-resc #Or whatever it may now be, this step is necessary, Makefile problem.

    cd /usr/src/linux-2.4.22-resc 
    make oldconfig           # You might have to answer some questions here. 
    make dep 
    make bzImage 
    make modules 
    make modules_install 
    make install

If the grub bootloader file has been changed (it will be under Redhat), then you may want to delete the entries for the new kernel from grub.conf.

3.3 Compiling the new cloop

Because we have compiled a new kernel we also need to replace a special module called cloop.o which is not generally part of the kernel distribution. The function of this module is to allow us to mount a compressed loopback filesystem. To this effect we need to compile a new cloop module. This part of the procedure is the one that caused the most headaches, trying to decide whether I was using cloop or gcloop - this information can be found in the original cloop.o using the strings command. This was due to a number of undocumented steps required to get the new module working, and also due to the choice of cloop module source code that we choose. The two version available on the WWW are:

The Knoppix cloop, found at [ http://www.knopper.net/knoppix/sources/]
gcloop from the Gentoo project, found at http://dev.gentoo.org/~lu_zero/gcloop/

The systemresccd reverted back to using the Knoppix cloop module as this was regarded to be the safest, and most stable, ath the time or writing, this is version 1.02.

The build is reasonably straightforward with a single Makefile change to the kernel version (strictly speaking this can be done one the commandline, without needing to modify the makefile):

3.3.1 Makefile modifications

There are some slight modifications that must be made:

KERNEL_DIR

  KERNEL_DIR=/usr/src/linux-2.4.22-resc

At this stage you should be able to make all targets in the Makefile. The generated module will be called "cloop.o"

3.4 Now save the kernel data

After installation the new kernel will be found in /boot/vmlinuz-2.4.22-resc. We can now use this kernel and the kernel modules found under /lib/modules/2.4.22-resc. We will save a copy of the kernel, the modules and the new cloop module onto the same USB/IDE disk where we generate our new rescue CD:

Plug in USB disk drive.

  mount -t ext3 /dev/sda3 /mnt/disk                     # Assuming the disk is recognised as sda 
  mkdir /mnt/disk/newkern 
  cd /mnt/disk/newkern;  
  tar cf kern-mod-2.4.22-rec.tar /lib/modules/2.4.22-resc   
  #  
  cp /path/to/new/cloop.o . 
  cp /boot/vmlinuz-2.4.22-resc .

3.5 Initial steps - Burn new CD

csh> su -  
sh# cdrecord -v dev=0,0,0 blank=fast 
sh# cdrecord -v dev=0,0,0 systemrescuecd-0.2.5.iso

3.6 Generate new Image file

This part of the procedure will be carried out on Machine B. During this phase, we are going to extract the data from our current bootable rescue CD to a writeable loopback filesystem, make some modifications and then save these changes.

When the machine B has booted from the rescue CD, the Bootloader manager should offer a list of options that you can provide to the kernel. I find it easier to choose vmlinuz1 nofb nonet, this will stop the boot process from trying to do fancy graphical output.

When the CD has booted and you are given a commandline prompt and root shell you can:

Connect the USB disk to the machine.
Once the device registers with the OS, you should be able to

    #Third disk partition, used to hold our customised image. 
    mount -t ext3 /dev/sda3 /mnt/temp3  
    cd /mnt/temp3/ 
    mkdir rescue 
    cd rescue 
    dd if=/dev/zero of=img25 bs=1M count=700   #This will be our image file 
    mke2fs -F -q -N 50000 img25 
    mount -t ext2 -o loop img25 /mnt/custom

The script for creating the custom CD /usr/sbin/sysresccd-custom will always refer to /mnt/custom. This is not changeable without editing the script. And since the script is on read-only media, we need to copy this script to a non read-only area like /tmp. Editing the script also turns out to be a necessity as there is an error in the script which will cause the script to die, this is due to a wrong parameter been given in an awk expression to determine the amount of disk space available on /mnt/custom (about line 145 - look for line involving df , change print $4 to print $3). This error is infact due to the long mountpoint name that I use (a shorter name like /tmp/a/img25 would not require script modification). I have contacted Francois about this, future versions should correct this minor flaw.

  cp /usr/sbin/sysresccd-custom /tmp

The same fix can be achived by changing the df -m to df -P -m.

Edit the line as mentioned above. Once this is done we can extract a copy of our bootdisk to /mnt/custom/customcd/files.

   /tmp/sysresccd-custom extract

Before making any changes to the image, it would be a good time to make a backup of the work done so far with the image file, this will give us a fallback copy of the work done so far if we mess up anything:

   cp /tmp/sysresccd-custom extract /mnt/temp3/rescue 
   umount /mnt/custom 
   cd /mnt/temp3/rescue/img25 /mnt/temp3/rescue/img25.bak

3.7 Customization of the filesystem

These are the steps that are required to implement the CD customisation.

  mount -t ext2 -o loop img25 /mnt/custom

Add entries for ttyS0,ttyS1,tts/0,tts/1 to /etc/securetty (One line per entry)
Change root password

    passwd; cp /etc/shadow /mnt/custom/customcd/files/etc/shadow

Change root shell from zsh to /bin/bash, if you do not make this change, you will not be able to log in on the ttyS console. I haven't discovered the source of this bug.
Add inittab entries to start up agettys on ttyS0 and/or ttyS1, also change zsh to bash (or disable tty0-6)

S0:2345:respawn:/sbin/agetty -h -t 60 ttyS0  115200  vt102 
S1:2345:respawn:/sbin/agetty -h -t 60 ttyS1  115200  vt102

disable dhcp and soundcard detection
Add in new kernel modules to /mnt/custom/files/lib/modules

   cd /mnt/custom/customcd/files 
   tar xpf  /mnt/temp3/newkern/kern-mod-2.4.22-rec.tar

3.8 Update isolinux.cfg

We need to add a new entry to our list of available kernels (infact we shall only have one option in the set of bootable kernels, this will have a very short timeout and will redirect to /dev/ttyS{0,1}. Unless your motherboard can redirect the BIOS over the serial port, you will never get a chance to choose the other kernels (nor should you really need to, although memtest is probably a good inclusion on the bootable CD).

The entry that I added is to /mnt/custom/customcd/isoroot/isolinux/isolinux.cfg is:

label serial 
        kernel vmlinuz2 
        append initrd=initrd2 nonet acpi=off root=/dev/ram0 init=/linuxrc setkmap=uk console=ttyS0,115200

Also add to line 1:

  serial 0 115200

and change default to serial. If you want, you can decrease to 0 (for immediate boot).

3.9 Update kernel

install the compiled kernel into isoroot

   cd /mnt/custom/customcd/isoroot/isolinux 
   cp /mnt/temp3/newkern/vmlinuz-2.4.22-resc ./vmlinuz1

3.10 Update initrd

update initrd to use only the UK keymap
replace the cloop module
replace the storage modules.

The ram disk inititialisation sequence must be updated to accomodate the new modules. This is found in the compressed (using gzip) loopback ext2 filesystem initrd1

  cd /mnt/custom/customcd/isoroot/isolinux 
  cp  initrd1 initrd1.bak 
  cp  initrd1 /tmp/initrd1.gz 
  cd /tmp 
  gunzip initrd1.gz 
  losetup /dev/loop1 initrd1 
  mkdir -p /mnt/loop1 
  mount /dev/loop1 /mnt/loop1 
  cp /mnt/temp3/newkern/cloop.o /mnt/loop1/cloop.o   # Overwrites old cloop.o 
  ### Comment on replacing other modules #### 
 
  cd /mnt/loop1/modules/storage 
  modules_list=`ls *.o` 
  cd /mnt/custom/customcd/files/lib/modules/2.4.22-resc/ 
  for i in $modules_list 
  do 
     find . -name $i -exec cp \{\} /mnt/loop1/modules/storage/ \; 
  done 
 
  cd /tmp 
  umount /mnt/loop1 
  losetup -d /dev/loop1 
  gzip -9 initrd1 
  cp initrd1.gz /mnt/custom/customcd/isoroot/isolinux/initrd1

3.10.1 Set the keymap

/tmp/sysresccd-custom setkmap uk

3.10.2 Generate a new cloop

From the previous steps we should be able to generate a new LiveCD which will boot into a serial console. This procedure generates the new livecd.cloop file in /mnt/custom/customcd/isoroot/.

  /tmp/sysresccd-custom cloop 250 20000

3.11 Generate the new ISO file

  /tmp/sysresccd-custom isogen GI_RESCCD

The new ISO is saved as /mnt/loop0/customcd/isofile/sysresccd-new.iso

3.12 Buring the new CD

From the previous steps we should be able to generate a new LiveCD which will boot into a serial console.

3.13 Caveats

On tty0 to tty6, the zsh shell seems to break. You can either change this to /bin/bash or disable them altogether in /etc/inittab.