So You Want A FreeBSD Debugging Kernel…
Installing a debugging kernel in FreeBSD
The Use Case
There are numerous reasons for needing a debugging kernel. For example, during a kernel panic. This was the case with us. Therefore, we installed a kernel with DDB enabled. In the event of a kernel panic, the system would drop to the debugger and the software engineers were then able to begin diagnosing the problem.
Preparing To Build The Kernel
Prerequisites
- FreeBSD source tree
- A kernel config
FreeBSD Sources
Operators can use cvs or Subversion to install the sources. Information on FreeBSD source trees can be found here.
The Kernel Config
Create a kernel config by opening /usr/src/sys/${arch}/conf/DEBUG. Where ${arch} is the machine architecture the kernel is to be built for. In this case, we build an amd64 kernel on an amd64 machine. The kernel config should appear as follows:
include GENERIC
options KDB_UNATTENDED
options DDB
Compiling and Installing The Kernel
Compiling and installing the debugging kernel is as simple as executing:
# cd /usr/src
# make buildkernel KERNCONF=DEBUG INSTKERNNAME=DEBUG
# make installkernel KERCONF=DEBUG INSTKERNNAME=DEBUG
Executing/Using The Kernel
From the loader menu
At the loader menu, press 6 to escape to the loader(8) prompt and type:
boot DEBUG
From the running system
The DEBUG kernel can be loaded by default by adding/modifying the kernel directive in loader.conf as follows. Each subsequent boot will load the DEBUG kernel.
kernel=DEBUG
Dropping into the debugger
During a kernel panic, it will simply reboot unless debug.debugger_on_panic is enabled. To enable this execute:
sysctl debug.debugger_on_panic=1
To induce a panic:
sysctl debug.kdb.panic=1
Installing FreeBSD 8.x Distributions After Installation
Installing Distributions After The Fact
FreeBSD 8.x [ and earlier ] media contain distributions one can select to install via sysinstall(8). This is accomplished via GUI or automated scripting requiring an install.cfg.

This image shows a typical FreeBSD installation where a kernel/distribution is extracted onto disk, What is an admin to do when they later find that they need to install a distribution after the fact? There are two methods to installing media distributions after the fact.
Using the Command Line
This is my preferred method as it does not require using sysinstall‘s graphical interface, but I am sure both have their merits. In this example, we install lib32. First, download the distribution from FreeBSD.org…
echo CHECKSUM.SHA256 CHECKSUM.MD5 install.sh lib32.a{a,b,c,d,e,f,g,h,i,j,k,l,m} lib32.inf lib32.mtree | xargs -n 1 -I%s fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/8.2-RELEASE/lib32/%s
The next step is to install the distribution…
./install.sh
and you’re done…
Using sysinstall(8)
Start the procedure by executing sysinstall. It opens an interactive GUI interface. The Following procedure installs the distribution…
Select “Configure”…

Select “Distributions”…

Select the distribution to install…

Select install media (this example uses ftp)…

Select ftp location (This step may differ depending on the media selected above)…

If networking is configured, select yet. Otherwise, select no and sysinstall will prompt for network configuration…

The distribution installs…

…and when completed, exit sysinstall.
References
* The command line used can be referenced in the FreeBSD Forums
Encrypting User Password Strings
Encrypting User Password Strings
The Use Case
User account passwords can be set during systems provisioning in multiple ways. Setting a host’s root password is a common use case. During a FreeBSD 8.x install, one can set root’s password (or other user account password) executing the following within the install.cfg:
/bin/sh -c ‘echo passwd |pw mod user username -h 0’
Unfortunately, doing so exposes the user’s account password. Therefore, the recommendation is to use an encrypted hash as follows:
/usr/bin/chpass -p $1$nPUexUs5$O4JuN.Ed/LqWHJKmf8K0h0 root
Encrypting The Password
The use case begs the question: How does one ascertain an encrypted hash given a string?
On FreeBSD, there are varying methods to ascertain an encrypted hash. Here are just two of the many.
openssl
# openssl passwd -1 MySecret
$1$YRth1v3T$MvGupL8n.VBjvM12JhR4G0
openssl(1) returns the entire encrypted hash. The character between the first and second dollar signs identifies the crypt mechanism. The following eight characters are the salt. The remaining characters following the third dollar sign are the encrypted string.
md5
# md5 -s MySecret
MD5 (“MySecret”) = 4132d75e6cb04073cc7756707057027f
md5(1) returns a partial encrypted hash. In the key-value pair, the key is the crypt mechanism. The first 8 characters of the value are the salt. The remaining characters are the encrypted string.
In this method, one must prepend $1$ to the value before feeding it into chpass(1).
Related Documentation
* Chapter 15, Security, of the FreeBSD handbook explains identifying a host’s crypt mechanism.
Automating Generation of the mfsroot.gz
NOTE: This post is old. I no longer generate mfsroots in this fashion and my methods continue to evolve, but my latest approach to modifying mfsroots is documented in the Installing FreeBSD via Cobbler post.
Automating Generation of the mfsroot.gz
Like the crunchgen(1) script, this is a starting point only that makes many assumptions. In fact, while writing this I have come up with plans to combine them.
I store an uncompressed mfsroot.gz file in subversion. It is stored in a manner that includes the files in stand/. You may or may not be aware that this is where the boot_crunch file exists. It also happens to be the location of all the hardlinks that point back to the boot_crunch inode. Subversion stores these as if they were all binaries. Below you will see that I programmatically restore the hardlinks.
A side effect of this management structure is that generating the same mfsroot.gz multiple times results in differing file sizes each time despite using the same content. I’m not 100% sure, but believe that this is due to the file metadata being different each time the content is exported from the subversion repo.
NOTE: The URL in the script below is a dummy URL. The real URL is internally hosted with no connectivity to the Internet.
#! /bin/sh
#
# mkmfsroot.sh: Generate a FreeBSD mfsroot file. This script does not
# function on hosts that are not running FreeBSD.
#
# Author: Rick
# Date: 21 August 2012
# Modif:
#####
## Variables
#####
SELF="${0##*/}";
NULL="/dev/null";
TMP="/tmp";
PATH="${PATH}";
ZERO="/dev/zero";
MFSROOTDIR="${1}";
SVNREPO="https://dummy.subversion.com/ats/ad/freebsd-prov/mfsroot/${MFSROOTDIR}";
REPODIR="/data/freebsd-prov/mfsroot/${MFSROOTDIR}";
MFSROOTSTAND="${REPODIR}/stand";
SVN="/usr/local/bin/svn";
STANDFILES="doconfig.sh install.cfg boot_crunch";
MFSROOT="/tmp/mfsroot";
MFSROOTMNT="/tmp/mfsrootmnt";
#####
## Functions
#####
# Clean what this script has done
CLEANUP() {
[ "${MDCFG}" = "True" ] && mdconfig -d -u ${DEVNUM};
if [ -d ${MFSROOTMNT} ]; then
cd && umount ${MFSROOTMNT} > ${NULL} 2>&1;
rm -rf ${MFSROOTMNT};
fi
[ -e ${MFSROOT} ] && rm ${MFSROOT};
[ -e ${REPODIR} ] && rm -rf ${REPODIR};
}
# We are erroring and exiting the script
ERR_EXIT() {
echo "${1}" 1>&2;
CLEANUP;
exit 1;
}
#####
## Main
#####
# Force two command line arguments
[ -z ${1} ] && ERR_EXIT "Specify an mfsroot!";
# Verify we're on a FreeBSD host
HOSTOS=`uname -s`;
[ ${HOSTOS} != "FreeBSD" ] && ERR_EXIT "Build host is not FreeBSD! Exiting!";
# Ensure we're using the latest repo version
[ -d ${REPODIR} ] && rm -rf ${REPODIR};
${SVN} export ${SVNREPO} ${REPODIR};
# Does the mfsroot dir we asked for exist?
[ ! -d ${REPODIR} ] && ERR_EXIT "${1} does not exist! Exiting!";
# Remove .svn if it exists
[ -d ${REPODIR}/.svn ] && rm -rf ${REPODIR}/.svn
# Fix stand/
#
# What do we do here and why?
#
# Since the contents of the mfsroot are stored in subversion, when the content
# is checked out or exported, each of the files in stand/ are stored as
# individual files when they are actually supposed to be hardlinks back to
# boot_crunch. Therefore, the code below checks out the content, removes the
# files in stand/, and exports boot_crunch and a couple other scripts. It
# finishes by regenerating all the hardlinks back to boot_crunch.
#
# We could resolve this in any number of ways, but as a stop gap measure for
# now we will leave it this way and fix it in the future.
if [ -d ${MFSROOTSTAND} ]; then
rm ${MFSROOTSTAND}/*;
for ea in ${STANDFILES}; do
${SVN} export ${SVNREPO}/stand/${ea} ${MFSROOTSTAND};
done
cd ${MFSROOTSTAND};
for i in $(./boot_crunch 2>&1 | grep -v usage); do
if [ "$i" != "boot_crunch" ]; then
rm -f ./"$i";
ln ./boot_crunch "$i";
fi
done
fi
# Calculate the size of the mfsroot, prepare the file and populate it
MFSROOTSIZE=$(du -sk ${REPODIR} | awk '{ print $1 }');
[ -z ${MFSROOTSIZE} ] && ERR_EXIT "Unable to determine mfsroot size! Exiting!";
dd if=${ZERO} of=${MFSROOT} bs=1024 count=$((${MFSROOTSIZE} + 1024));
[ $? -ge 1 ] && ERR_EXIT "Unable to create ${MFSROOT}! Exiting!";
MFSROOTDEV=$(mdconfig -f ${MFSROOT});
[ -z ${MFSROOTDEV} ] && ERR_EXIT "Unable to mount ${MFSROOT}! Exiting!";
DEVNUM=$(echo ${MFSROOTDEV} | awk '{ print substr($0,length($0)-0,length($0)) }'
);
MDCFG="True";
newfs /dev/${MFSROOTDEV};
[ ! -d ${MFSROOTMNT} ] && mkdir ${MFSROOTMNT};
mount /dev/${MFSROOTDEV} ${MFSROOTMNT} && \
(cd ${REPODIR}; tar -cf - .) | (cd ${MFSROOTMNT}; tar -xf -);
# Now the new mfsroot has been populated, it's time to save it and clean up
umount ${MFSROOTMNT} && mdconfig -d -u ${DEVNUM} && gzip ${MFSROOT};
[ $? -ne 0 ] && ERR_EXIT "Error unmounting ${MFSROOT}! Exiting!";
MDCFG="False";
CLEANUP;
#####
## EOF
#####
FreeBSD uname(1) and the Environment
FreeBSD uname(1) and The Environment
An environment variable composed of the string UNAME_ followed by any flag to the uname utility (except for -a) will allow the corresponding data to be set to the contents of the environment variable.
I would like to call out the environment variables specifically and identify what is changed when these environment variables are utilized. They are as follows:
- UNAME_s: The name of the operating system implementation
- UNAME_n: The name of the system
- UNAME_r: The current release level of the operating system
- UNAME_v: The version level of this release of the operating system
- UNAME_m: The type of the current hardware platform
- UNAME_p: The type of the machine processor architecture
- UNAME_i: The kernel ident
GNU tar Header Errors?
Dealing with GNU/tar’s Extended Header Warnings

It seems that FreeBSD‘s tar binary utilizes additional headers that GNU tar does not recognize. There are many blog posts about this. The problem appears to affect tarballs created on various operating systems that employ varying versions of tar and extracting those tarballs with GNU tar.
Generally speaking, these warning can be ignored. On the command line, the files contained within the tarball will still extract (or at least they have every time I’ve encountered this). This does, however, present a problem if one is programmatically encountering this. One can resolve this by addressing the issue within their code or by using the same version of tar on each platform to create and extract the tarballs in question.
Automating FreeBSD’s crunchgen(1)
Automating FreeBSD’s crunchgen(1)
One of my main tasks in my role at work is to maintain FreeBSD provisioning bits. I have found myself generating boot_crunch files with various binaries and what not to extend the capabilities of the provisioning aspect of FreeBSD. So much so that I whipped up a quick script to automate the task for me.
Please bear in mind, it is a quick script which makes a number of assumptions. It’s not perfect and can certainly use some tweaking and the like, but this initial version gets one started.
I keep my boot_crunch.conf in a subversion repo, thus the calls to svn.
#! /bin/sh
#
# bldcrunch.sh: Generate a FreeBSD boot_crunch file. This script does not
# function on hosts that are not running FreeBSD.
#
# Author: Rick
# Date: 25 July 2012
# Modif:
#####
## Variables
#####
SELF="${0##*/}";
NULL="/dev/null";
TMP="/tmp";
PATH="${PATH}";
SVNREPO="http://svn.domain.com/freebsd-repo";
REPODIR="/freebsd-repo";
BOOTCRUNCHDIR="boot_crunch";
BOOTCRUNCHCONF="boot_crunch.conf";
BOOTCRUNCHMK="boot_crunch.mk";
SVNBIN="/usr/local/bin/svn";
CRUNCHGEN="/usr/bin/crunchgen";
MAKE="/usr/bin/make";
#####
## Functions
#####
ERROR_EXIT() {
echo "${1}" 1>&2;
[ -d ${TMP}/${BOOTCRUNCHDIR} ] && rm ${TMP}/${BOOTCRUNCHDIR};
exit 1;
}
#####
## Main
#####
# Verify we're on a FreeBSD host
HOSTOS=`uname -s`;
[ ${HOSTOS} != "FreeBSD" ] && ERROR_EXIT "Build host is not FreeBSD! Exiting!";
# Do all of our binaries exist?
for ea in ${SVNBIN} ${CRUNCHGEN} ${MAKE}; do
if [ -e ${ea} ]; then
continue;
else
ERROR_EXIT "${ea} is required! Exiting!";
fi
done
# Update SVN repo to make sure we're using the latest boot_crunch.conf
if [ -d ${REPODIR}/${BOOTCRUNCHDIR} ]; then
cd ${REPODIR}/${BOOTCRUNCHDIR} && ${SVNBIN} update;
else
${SVN} co ${SVNREPO} ${REPODIR};
fi
# Setup our build environment
[ -d ${TMP}/${BOOTCRUNCHDIR} ] && rm -rf ${TMP}/${BOOTCRUNCHDIR};
mkdir ${TMP}/${BOOTCRUNCHDIR} && cd ${TMP}/${BOOTCRUNCHDIR};
# Build the boot_crunch
${CRUNCHGEN} -o ${REPODIR}/${BOOTCRUNCHDIR}/${BOOTCRUNCHCONF};
[ $? -eq 0 ] && ${MAKE} -f ${BOOTCRUNCHMK};
[ $? -ne 0 ] && ERROR_EXIT "Error generating boot_crunch! Exiting!";
#####
## EOF
#####
BTW, I edited this because I found out about wordpress’s sourcecode tag…awesome!
FreeBSD 8.x and 4K sector disks
FreeBSD 8.x and 4K sector drives
FreeBSD 8.x (and older) use sysinstall as the install mechanism and fdisk for disk slicing/partitioning. Unfortunately, these utilities perform calculations based on a 512B sector size. This causes problems with drives of a sector size greater than this. 4K sector size drives are the most popular at this point.
I cannot explain the problem and solutions any better than Ivan Voras (FreeBSD source contributor since 2008), so I link to his blog here.
There are good tidbits of information in the comments such as sysinstall not understanding gpart, etc. It would be a good idea to read through the comments as well.
FreeBSD + Dell 12G + X520-DA2 10G NIC
Intel 82599 based NICs on Dell 12G Servers w/ FreeBSD?
Dell offers the X520-DA2 10G Fiber NIC based on the Intel 82599 chipset for it’s 12G line of servers. It seems that there were two batches of this NIC manufactured. One batch was Dell branded, the other Intel branded.
According to this post, the FreeBSD source appears to be missing PCI ID’s etc that would enable the Dell branded card to work under FreeBSD on it’s 12G servers resulting in the card not being recognized by the OS. Inspection of dmesg shows that there is an ethernet card with no driver attached.
This is currently being corrected in HEAD, but for the time being swapping in an Intel branded equivalent which is seen by FreeBSD without a problem is sufficient. Unfortunately, the Dell BIOS does not recognize the card. having said that, the card is recognized in the OS and does work using the ixgbe(4) driver. The interface name begins with “ix”.