Diskless Linux with PXE HOWTO

You'll find all downloads and the current version of this document at http://www.intra2net.com/de/produkte/opensource/diskless-howto/

If you have comments or additions to this HOWTO don't hesitate to write to .

Revision: 1.1 (2004-02-05):

• Fix trim function in bootimage
• Add fsck workaround
• Add retries to dhcp client (thanks to Joe Robertson)
• Add SuSE comment

Revision: 1.0 (2003-12-21): Original release

1. Legal stuff

2. Pre-Requisites

• a diskless client with PXE remote boot capabilities. Most on-board network cards have a PXE client in BIOS. The more expensive NICs often come with their own small PXE BIOS. Sometimes you have to activate it first - I had to use IBAUtil to activate PXE on my Intel EEPro/100+.
• a Linux distribution CD you want to run on the diskless system
  
• a DHCP server (e.g. dhcpd that comes with most distributions)
• a TFTP server (e.g. tftp-server that comes with most distributions)
• a NFS server
• DHCP, TFTP and NFS can live happily together on one server

3. Installing the distribution

1. Install the distribution on a disk and then copy all files into a directory onto your server
2. Install the distribution directly into a dir on the server. I have a script to install a minimal Fedora Core 1. Mount the first Fedora CD, create an empty directory and run it like this: ./mkfedora.sh [TARGETDIR] [CDROMPATH]

# we'll have a read-only /etc
ln -sf /proc/mounts ${SYSROOT}/etc/mtab
ln -sf /var/resolv.conf ${SYSROOT}/etc/resolv.conf

# replace the entry for / in fstab: rc.sysinit otherwise tries to do a fsck.ext2...
cp ${SYSROOT}/etc/fstab ${SYSROOT}/etc/fstab.orig
grep -v -e "[ 	]/[ 	]" ${SYSROOT}/etc/fstab >${SYSROOT}/etc/fstab.new
echo "none			/			tmpfs	defaults	0	0" >>${SYSROOT}/etc/fstab.new
mv -f ${SYSROOT}/etc/fstab.new ${SYSROOT}/etc/fstab

# disable networking (we have it already set up if the rc's are running)
/usr/sbin/chroot ${SYSROOT} /sbin/chkconfig --del network

# disable kudzu (won't do anything since we are on a read-only filesystem)
/usr/sbin/chroot ${SYSROOT} /sbin/chkconfig --del kudzu

# move RPM database to /usr (saves ram since /var will live in tmpfs)
mkdir -p ${SYSROOT}/usr/var/lib/
mv ${SYSROOT}/var/lib/rpm/ ${SYSROOT}/usr/var/lib/
ln -s /usr/var/lib/rpm ${SYSROOT}/var/lib/rpm

# remove temp files from RPM database
rm -f ${SYSROOT}/usr/var/lib/rpm/__db.*

# compress /dev to speed up booting
tar czf ${SYSROOT}/dev.tar.gz -C ${SYSROOT} dev

4. Configuring NFS

/netboot/pxeclient  192.168.1.0/24(ro,no_root_squash)

5. Prepairing the initrd image

# unzip and mount base image
gunzip ${IMAGE}.gz
mkdir __pxeboot-tmp__
mount -o loop $IMAGE __pxeboot-tmp__
[ $? -ne 0 ] && echo "error mounting image" && exit 1

# create module directories
mkdir -p __pxeboot-tmp__/lib/modules/${KERNELVER}/net
mkdir -p __pxeboot-tmp__/lib/modules/${KERNELVER}/nfs

# recalculate dependencies
/sbin/depmod -a -F $SYSTEMMAP -b __pxeboot-tmp__ -C /dev/null $KERNELVER

# unmount and zip again
umount __pxeboot-tmp__
gzip -9 $IMAGE
rmdir __pxeboot-tmp__

6. Configuring DHCP

host pxeclient {
   hardware ethernet AA:BB:CC:DD:EE:FF;
   fixed-address 192.168.1.55;
   option host-name "pxeclient";

   filename "/pxelinux.0";
   next-server tftpserver;
   option root-path "nfsserver:/netboot/pxeclient";
}

7. Configuring TFTP

LABEL linux
        KERNEL vmlinuz-2.4.22-1.2115.nptl
        APPEND initrd=pxeboot.img.gz ramdisk_size=8192

8. Booting

9. Reducing boot times

• using a custom kernel. Throw everything out you don't need. My experience is that especially IDE is taking a lot of time to find out that I don't have any hard disk connected. You'll need to have support for your network card, initrd images, NFS, ext2fs and tmpfs.
• Tweak your /etc/rc.d/rc.sysinit. Usually it is doing stuff like fsck and depmod that take time but don't help anything since your are on a read-only image. Remove everything you don't need.
  

10. Known problems

• Shutdown doesn't work: this is due to the client trying to unmount it's nfs root. I haven't found a distribution-independent way to fix this. If you want to try it: you've got to make sure that the root nfs isn't unmounted and the portmapper (if running) isn't killed. Please send me your solutions and I'll add them.
  
• No DHCP client daemon running: The system isn't running a DHCP daemon to renew its lease. This isn't a problem if you use fixed addresses with DHCP like in the example above. But if you are using a DHCP pool your client might get thrown out and the IP reassigned if the lease has expired. I didn't want to keep the daemon from the initrd alive since that would mean that we can't unmount initrd and will waste the 8 MB RAM. Starting a new dhclient from the real distribution is possible, but you usually have to tweak dhclient-script to make sure it isn't switching off the network device or changing the IP during discovery.

11. Differences for SuSE Linux

• SuSE kernels usually have nfs compiled in so you don't need to add it as module. But the af_packet module is needed for dhcp. You'll need to change mkbootimage (or do it by hand) and linuxrc (in pxeboot.img.gz).

Appendix A. The initrd image

#!/bin/sh

# (C) Copyright 2003,2004 by Gerd v. Egidy 
#
# PCI autodetect Copyright 2001-2003 by Diego Torres Milano
# PCI autodetect adapted from PXES by Gerd v. Egidy
# 
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 2 as 
# published by the Free Software Foundation.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#

autodetect_pci() # PCICLASS MODCLASS
{
    local PCICLASS NOTFOUND line

    PCICLASS=$1
    NOTFOUND=1

    if [ -z "$PCICLASS" ]; then
        echo "$PROGNAME: autodetect_pci missing parameter" >&2
        exit 1
    fi

    if [ ! -r /proc/bus/pci ]; then
        return $NOTFOUND
    fi

    # I can't make a pipe of this because ash treats it as a subshell,
    # that's why the temp file
    /sbin/lspci -i /dev/null -knm | /bin/grep "$PCICLASS" > /tmp/pci.$$
    while read line
    do
        set -- $line
        eval mod=\$$#
        eval mod=$mod
        if [ "$mod" != "UNKNOWN" ]; then
            echo "loading $mod..."
            /sbin/modprobe $mod
            NOTFOUND=0
        fi
    done < /tmp/pci.$$
    /bin/rm -f /tmp/pci.$$

    return $NOTFOUND
}

echo "Remount / read-write"
mount -n -o remount,rw /

echo "Mounting /proc filesystem"
mount -n -t proc /proc /proc

echo "Autodetecting network devices..."
autodetect_pci 'Class 02..'
if [ $? -ne 0 ]; then
    echo "WARNING: no network interface found"
fi

echo "Loading nfs modules"
modprobe nfs

echo "Initializing network loopback device"
ifconfig lo 127.0.0.1 up
    
echo "Configuring eth0"
udhcpc --now --quit --interface=eth0 --script=/bin/udhcpc.script
if [ $? -ne 0 ]; then
    echo "ERROR: couldn't get DHCP lease, trying again"
    udhcpc --now --quit --interface=eth0 --script=/bin/udhcpc.script
    if [ $? -ne 0 ]; then
        echo "ERROR: couldn't get DHCP lease, trying again"
        udhcpc --now --quit --interface=eth0 --script=/bin/udhcpc.script
        if [ $? -ne 0 ]; then
            echo "ERROR: can't get DHCP lease"
            exit 0
        fi
    fi
fi

# load DHCP parameter
. /etc/udhcpc-eth0.info

echo "Mounting nfs root filesystem"
if ! echo $ROOTPATH | grep -q ":/" ; then
    # we haven't got a full path, use next-server
    ROOTPATH="${NEXTSERVER}:${ROOTPATH}"
fi

if echo $ROOTPATH | grep -q "," ; then
    # we have options
    NFSOPTIONS=`echo $ROOTPATH | sed -e "s/\(.*\)\(,.*\)/\2/"`
    ROOTPATH=`echo $ROOTPATH | sed -e "s/\(.*\)\(,.*\)/\1/"`
fi

echo "Mounting root filesystem"
mount -rw -t tmpfs none /sysroot/

echo "Mounting NFS root-base"
mkdir /sysroot/nfsroot
mount -n -o "ro,nolock${NFSOPTIONS}" -t nfs "$ROOTPATH" /sysroot/nfsroot
if [ $? -ne 0 ]; then
    echo "ERROR: can't mount root filesystem via NFS"
    exit 0
fi

echo "Setting root symlinks"
cd /sysroot
find ./nfsroot -maxdepth 1 -mindepth 1 -exec ln -s \{\} \;
cd /

echo "Handling special directories"
rm -f /sysroot/initrd
mkdir /sysroot/initrd
rm -f /sysroot/tmp
mkdir /sysroot/tmp
rm -f /sysroot/proc
mkdir /sysroot/proc

echo "Copying /var"
rm -f /sysroot/var
mkdir /sysroot/var
cp -a /sysroot/nfsroot/var /sysroot
cp /etc/resolv.conf /sysroot/var/resolv.conf

# /dev handling
rm -f /sysroot/dev
mkdir /sysroot/dev
if [ -f /sysroot/nfsroot/dev.tar.gz ]; then
    echo "Unpacking /dev"
    tar xzf /sysroot/nfsroot/dev.tar.gz -C /sysroot
else
    echo "Copying /dev"
    cp -a /sysroot/nfsroot/dev /sysroot
fi

echo 0x0100 > /proc/sys/kernel/real-root-dev

echo "Unmounting temporary mounts"
umount /proc

echo "Changing to new NFS root"
pivot_root /sysroot /sysroot/initrd