Archive

Archive for May, 2012

FreeBSD on Dell PowerEdge 12G Servers

May 22, 2012 Leave a comment

The Problem

We received a number of the new Dell PowerEdge 12G servers and were excited to give them a try. We quickly found a scenario identical to the issue Wayfair Engineering discovered. The issue was that despite the hardware notes stating the LSI Logic SAS2208 chipset was supported (by an odd driver given past precedent) the installation DVD failed to recognize the controller and therefore failed to install FreeBSD.

The Hardware

We were testing FreeBSD 8.3-RELEASE on the following configuration:

Dell PowerEdge R620 w/ H710 RAID Controller
128GB RAM
Eight (8) 2.5″ 500GB SAS HDD

Dell PowerEdge R620

The Resolution

Upon discovery of this issue, I was on the path to resolve it within our environment in the same manner that Wayfair Engineering had done. The difference was that we would be working with stable/8 vs. stable/9.

While working on this approach, we were able to come into contact with FreeBSD contributor Sean Bruno. Sean indicated to us that the mfi driver changes had been committed to stable/8 on 18 May 2012, the week we had attempted to get FreeBSD to install on the R620. You can see the details of this commit here. Obviously, with this commit, it eliminated the need to download the MFI head project that Wayfair Engineering had to download to get it to work.

The mfi commit was done for stable/7, 8, and 9 on 18 May 2012. Therefore, if you are installing FreeBSD on your Dell PowerEdge 12G servers, you should be running one of those versions or a production release with that code.

The Process

# csup stable-supfile
# cd /usr/src/
# make buildworld
# cd /usr/src/release
# make release

You’ll want various environment variables set when you execute `make release` such as CHROOTDIR, CVSROOT, and BUILDNAME. You may also perhaps want to set MAKE_DVD, NO_FLOPPIES, NODOC, etc. Check the release(7) manpage for more information.

ISO files will be stored in $CHROOTDIR/R/cdrom/. Use `mkisofs` from cdrtools to generate your ISO image and burn it to CD/DVD using your favorite burner and software.

My environment variables were set as follows when I built the release:

CHROOTDIR=/usr/release
BUILDNAME=8.0-STABLE
CVSROOT=/usr/cvs
EXTPORTSDIR=/usr/ports
EXTSRCDIR=/usr/src
MAKE_DVD=YES
NO_FLOPPIES=Yes
NODOC=YES

Advertisements
Categories: FreeBSD

Using sysinstall for automated FreeBSD 8.x installs

May 8, 2012 Leave a comment

First, allow me to point out that as of FreeBSD 9 sysinstall is no longer the default install mechanism.  Therefore, this post addresses versions previous to FreeBSD 9.

Summary

It’s important to understand that our provisioning platform is Cobbler. Therefore, you may find references to Cobbler in this post. There may need to be code modifications to permit these concepts to work within other platforms.

This set of config files and scripts allows us to more dynamically control host builds by scripting much of the process and hosting an answer file remotely. Doing so helps us avoid the need of creating multiple mfsroot files for varying profiles, etc.

Out of the box, sysinstall does not support pure HTTP installs. To enable full HTTP support, one of my former colleagues wrote a patch which was applied to the sysinstall source and compiled. This new binary was put into the mfsroot file.

mfsroot.gz

The mfsroot.gz is a BSD UFS filesystem environment that contains config files, scripts, and binaries necessary to complete a FreeBSD 8.x install. This file is retrieved by the PXE boot file and loads it into memory. The procedure described below takes the contents of an mfsroot.gz from the FreeBSD DVD install media and adds the files described later in this post to that file.

The mfsroot contained within the vendor install media is setup to run sysinstall. It can be
modified in various ways to manipulate the FreeBSD install.

Prerequisites

  • mfsroot file: This file can be found on the vendor media. Just copy it
    locally.
  • A FreeBSD host: This host is where you will perform these procedures.
  • The FreeBSD host should have bash installed and when executing this
    procedure, it is the current shell

Procedure

Modifying the mfsroot involves changes to the existing file to change the behavior of the environment. Some common files modified are the loader.rc and loader.conf files. When adding new files to the mfsroot, such as a newly compiled boot_crunch file, it will require creating a new mfsroot file. This section will describe both methods of modifying the mfsroot.

Modifying Existing mfsroot

To modify an existing mfsroot all that needs to be done is to create a memdisk device and mount it. Once mounted, you can modify the mfsroot simply by cd’ing into the new mount point and making your desired changes. We assume the memdisk device is md0 and the mount point is /mnt.

To create this memdisk and mount it, one only need to run the following:

# mdconfig -f $path_to_mfsroot
# mount /dev/md0 /mnt

Once changes have been made to the mfsroot, you can dismount and remove the memdisk device by executing the following:

# umount /mnt mdconfig -d -u 0
# mdconfig -d -u 0

Creating New mfsroot from Existing mfsroot

When adding files to an mfsroot, such as the boot_crunch, will require a new, larger mfsroot. This procedure discussed how to accomplish this.

This procedure assumes the following:

  • we are adding a boot_crunch file
  • Resulting mfsroot will need to be 12MB.
  • Existing mfsroot mount point will be /tmp/mfsroot_old
  • New mfsroot mount point will be /tmp/mfsroot_new
  • Existing mfsroot is /tmp/mfsroot_old
  • Memdisk device for the existing mfsroot is md0
  • Memdisk device for the new mfsroot is md1
  • New boot_crunch file is /tmp/bootcrunch/boot_crunch
# mkdir /tmp/mfsroot_old /tmp/mfsroot_new
# dd if=/dev/zero of=/tmp/mfsroot bs=1024 count=12288
# mdconfig -f /tmp/mfsroot_old; mdconfig -f /tmp/mfsroot
# newfs /dev/md1
# mount /dev/md0 /tmp/mfsroot_old; mount /dev/md1 /tmp/mfsroot_new
# cd /tmp/mfsroot_old; tar -cf - . | (cd/tmp/mfsroot_new; tar -xf -)
# cp /tmp/bootcrunch/boot_crunch /tmp/mfsroot_new/stand/
# cd /tmp/mfsroot_new/stand
# for i in $(./boot_crunch 2>&1|grep -v usage); do
    if [ "$i" != "boot_crunch" ]; then
      rm -f ./"$i"; ln ./boot_crunch "$i";
    fi
  done
# cd /; umount /tmp/mfsroot_old; umount /tmp/mfsroot_new
# mdconfig -d -u 0; mdconfig -d -u 1

At the completion of this procedure, /tmp/mfsroot will be the new mfsroot. Replace the old mfsroot with this one.

install.cfg

The install.cfg file is sysinstall’s config file. When sysinstall is executed it checks for the existence of this file. If the file does not exist, it runs interactively else it executes the file and performs the operations contained within it. This file exists in stand/ inside of the mfsroot file.

The syntax of the file is archaic and strict, but can be quite powerful if you spend the time to learn it. That is a moot point now though since sysinstall is deprecated in more recent versions of FreeBSD.

The file below simply sets up the environment where sysinstall will run, then executes a script called doconfig.sh which is described below.

# Turn on extra debugging.
debug=YES

# Turn off all prompting.
nonInteractive=YES
noWarn=YES

command=/bin/sh /stand/doconfig.sh
system
# Chain to the config we just downloaded
configFile=/stand/cobbler.cfg
loadConfig

doconfig.sh

This script, also in stand/ inside the mfsroot file, is the workhorse. It completes the setup of the environment by setting variables and communicating with the remote server to grab the remainder of the install.cfg file which we see is referred to as stand/cobbler.cfg. We describe this later in the post.

#!/bin/sh

server=`kenv -q boot.nfsroot.server`
mac=`kenv -q boot.netif.hwaddr`
ip=`kenv -q boot.netif.ip`
nm=`kenv -q boot.netif.netmask`
gw=`kenv -q boot.netif.gateway`
name=`kenv -q dhcp.host-name`
route=`kenv -q dhcp.routers`

# Returns true if a given network interface has the specified MAC address.
macmatch()
{
  local addr

  addr=`ifconfig $1 | grep ether | awk -F" " '{ print $2 }'`
  [ "$addr" = "$2" ]
  return $?
}

for ifn in `ifconfig -l`; do
  case $ifn in
    *)
      if macmatch $ifn $mac; then
        iface=$ifn
      fi
    ;;
  esac
done

# Bring up the interface so we can fetch the remote install.cfg file.  Confuses sysinstall, so
# bring it back down after fetching install.cfg.  Kinda assuming we're on the same subnet
# as the server, otherwise won't work.
ifconfig "$iface" "$ip" netmask "$nm"
sleep 5
route add default $gw

# Use Fetch to get my answer file from my cobbler/remote server.  Use awk to pull out different
# sections using "% /path/to/file" syntax.
fetch -qo - "http://$server/cblr/svc/op/ks/system/$name" |
	awk '/^% /{f=$2} /^[^%]/ && f{print > f}'

# Bringing down the interface
ifconfig $iface down
route delete default $gw

# Setup our media.  This file is loaded by sysinstall after it retrieves the install.cfg from the
# cobbler/remote server.
cat > /stand/media.cfg <<EOF
netDev=${iface}
defaultrouter=${route}
ipaddr=${ip}
netmask=${nm}
_httpPath=http://${server}/cobbler/ks_mirror/freebsd82-x86_64
mediaSetHTTP
EOF

cobbler.cfg

The cobbler.cfg is hosted on a remote server and is fetched via HTTP. It can be separated into sections that are executed as shell scripts to enable more dynamic control over the configuration. The doconfig.sh script above separates the sections of the files based on “% $filename” syntax.

The $disk variables below are substituted with the actual disk identifier. You can find that using the install.disk kernel variable.

% /stand/cobbler.cfg
# The installation media is setup in the doconfig.sh script
hostname=$system_name
configFile=/stand/media.cfg
loadConfig

# Select which distributions we want.
dists=base kernels GENERIC SMP doc catpages
distSetCustom

# Figure out the disk configuration
disk=${disk}
partition=all
bootManager=standard
diskPartitionEditor
${disk}s1-1=ufs 12582912 / # 6 GB root
${disk}s1-2=swap ${swap} none # swap
${disk}s1-3=ufs 2097152 /tmp 1 # tmp
${disk}s1-4=ufs 4194304 /var 1 # 2 GB var
${disk}s1-5=ufs 4194304 /home 1 # 2 GB home

# OK, everything is set.  Do it!
installCommit

package=perl
packageAdd

shutdown
Categories: FreeBSD

PXE Booting Into A FreeBSD Installation

May 4, 2012 3 comments

NOTE: This post is old. For our new installation method, see the article on Installing FreeBSD via Cobbler.

In the modern day, technologies for provisioning OS installations has progressed to a point where network installs are the standard thus eliminating the need for physical media and even physical access to the asset (apart from it’s physical installation in a cabinet). Networked consoles enable system builds in place.

Installing via PXE requires three services…DHCP, TFTP, and FTP or HTTP.  For the purposes of this post, we use ISC DHCP and the system TFTP daemon.  It is assumed that your FTP and/or HTTP servers are already configured appropriately.

The environment I installed in included Cobbler, a linux build system that has been modified by some of my colleagues to support FreeBSD.  Cobbler utilizes pxelinux as the PXE loader, therefore, my scenario included chaining Grub2pxe to pxelinux.  I have not tested without pxelinux, but I assume that Grub2pxe will work find without pxelinux.  Since we do employ Cobbler, my post will include the pxelinux configuration, but I will keep the information generic.

DHCP Configuration

An example of the dhcpd.conf file:

subnet 192.168.1.0 netmask 255.255.255.0 {
   option routers             192.168.1.1;
   option ntp-servers 192.168.1.1;
   option domain-name-servers 192.168.1.2, 192.168.1.3;
   option subnet-mask         255.255.255.0;
   pool {
      range 192.168.1.100 192.168.1.200;
   }
   filename                   "/pxelinux.0";
   default-lease-time         21600;
   max-lease-time             43200;
   next-server                192.168.1.5;
}

TFTP Configuration

An example of /tftpboot/pxelinux.cfg/default file:

LABEL bsdpxegrub
   kernel /tftpboot/pxegrub.0
   MENU LABEL bsdpxegrub
   append initrd=/tftpboot/pxegrub.0  locale=  text
   ipappend 2

Most of the above configuration is superfluous, but does not cause any issues with the FreeBSD install.

Grub2pxe Configuration

Grub2 is necessary because it includes support for the BSD UFS Filesystem of which the mfsroot.gz is. Therefore, download and install Grub2 as per the instructions in the Grub2 manual.

The files and directories of interest here are:

  • /tftpboot/pxegrub.0  (The PXE executable)
  • /tftpboot/boot/grub/   (The default Grub directory)
  • /tftpboot/boot/grub/grub.cfg   (The default Grub config file)
  • /tftpboot/boot/grub/i386-pc/*   (All of Grub’s modules)

Once Grub is installed, grub-mknetdir will put everything into place including running grub-mkimage which generates the pxegrub.0 file.  An example of the command is as follows:

grub-mknetdir –net-directory=/tftpboot –grub-mkimage=pxegrub.0

What you see below for the /tftpboot/boot/grub/grub.cfg, is a complete config for our environment. Other implementations may not need the config that we have generated. Many of the kernel environment variables below are used in our mfsroot.gz file. For a minimal working environment, all the lines up to and including the vfs.root.mountfrom kernel variable are necessary.

menuentry $arbitrary_profile_name {
   echo -e "Fetching the kernel and UFS root...\c"
   kfreebsd $kernel_path
   kfreebsd_loadenv /boot/device.hints
   kfreebsd_module $initrd_path type=mfs_root

   set kFreeBSD.vfs.root.mountfrom=ufs:/dev/md0c
   set kFreeBSD.boot.nfsroot.server=$pxe_default_server
   set kFreeBSD.boot.netif.hwaddr=$net_pxe_mac
   set kFreeBSD.boot.netif.ip=$net_pxe_ip
   set kFreeBSD.boot.netif.netmask=255.255.255.0
   set kFreeBSD.boot.netif.gateway=$system_gateway
   set kFreeBSD.dhcp.host-name=$net_pxe_hostname
   set kFreeBSD.dhcp.routers=$system_gateway
   set kFreeBSD.boot.kspath=$ks_path
   echo "Done!"
   sleep 2
}

HTTP/FTP Configuration

The HTTP/FTP configuration is beyond the scope of this post.  The assumption is made that FreeBSD is either mirrored locally or being accessed at freebsd.org remotely and that the install.cfg inside the mfsroot.gz is properly set to account for the media type and location.  An example of the variables would be mediaSetFTP accompanied by _ftpPath being set to something like ftp://ftp.freebsd.org/pub/FreeBSD/.  Note that using mediaSetFTP has several non-optional variables that must also be set.  Check the sysinstall manpage for more information.

Pure HTTP installs are not supported by sysinstall.  A former colleague wrote an HTTP module for sysinstall that we compiled into the binary that is placed inside the mfsroot.gz.  This patch was contributed back to the community, but was not committed to the FreeBSD sources due to the migration away from sysinstall in FreeBSD 9.0.

FreeBSD mfsroot Configuration

The actual contents and configurations of the mfsroot.gz are beyond the scope of this post, but are covered in this post.  The most important file, however, is the install.cfg that sysinstall consults upon boot of the environment.  Check the sysinstall manpage for complete information on generating an install.cfg for your environment.
Categories: FreeBSD