mfsroot starts sysinstall how?

December 18, 2012 Leave a comment

mfsroot starts sysinstall how?

I’ve been compelled to attempt to ascertain how sysinstall(8) is started from mfsroot[.gz] on multiple occasions. For various reasons, I never actually discovered the answer until I eventually encountered a scenario where my interest was peaked…I broke down and emailed freebsd-questions. I’d like to thank Jeremy for his response below!

It’s spawned by the kernel. Really. You’re probably used to the kernel
spawning /sbin/init, except that doesn’t exist in this environment. And
you probably don’t see the message about it on VGA console because
screen gets cleared immediately by sysinstall(8) right after it’s
printed. On serial console it’s much more evident — look closely at
the output (read slowly) and note the line right after “Trying to mount

The loader(8) variable called init_path defines a colon-delimited list
of executables for the kernel to attempt to run. The default value for
init_path is hard-coded into /boot/kernel/kernel. You can see it with
strings(1) if you want, but it defaults to this:


None of those, sans /stand/sysinstall, exist under mfsroot. The rest
you can figure out yourself. 🙂

You can override this by setting something [http_path] in loader.conf or by changing
the default in the kernel yourself.

I added init the boot_crunch file and the image started init(8) as opposed to sysinstall(8).


This blog is posted for informational purposes only. Extensive testing is recommended prior to implementing changes discussed here.

Categories: FreeBSD

FreeBSD Ports Batch Install

December 10, 2012 2 comments

FreeBSD Ports Batch Install

Many ports interactively prompt for various configuration options when installing. This is both annoying and time consuming. Fortunately, it can also be avoided.

`make config-recursive` interactively prompts users for options prior to `make install` or `make package-recursive`. This, however, is still interactive. Alternatively, one could add ‘-DBATCH’ to the command and install or create packages accepting default configuration options.

The following examples install and create packages including dependencies, respectively. In the second example, packages will be written to /usr/ports/packages if it exists.

# cd /usr/ports/sysutils/puppet
# make -DBATCH install


# cd /usr/ports/sysutils/puppet
# make -DBATCH package-recursive


This blog is posted for informational purposes only. Extensive testing is recommended prior to implementing changes discussed here.

Categories: FreeBSD

FreeBSD Partitions and Filesystems With GPART

December 3, 2012 3 comments

FreeBSD Partitions and Filesystems With GPART

The Problem

Installing FreeBSD 8.x on a machine with two disk volumes of 100GB and slightly more than 3TB. The quantity of cylinders on the 3TB volume is greater than the 65,535 cylinders supported by bsdlabel. Therefore, the filesystem does not consume the total available disk space.

This is resolved by installing and implementing gpart. I won’t go into the procedure here as Warren Block blogs on this topic very well at

Installing gpart

GPART is easily installed from package repos using pkg_add(1) as follows:

# pkg_add -r gpte

It may also be installed from ports by executing the following:

# cd /usr/ports/sysutils/gpte
# make install clean


This blog is posted for informational purposes only. Extensive testing is recommended prior to implementing changes discussed here.

Categories: FreeBSD

FreeBSD: Computing kern.maxfilesperproc and kern.maxfiles

November 12, 2012 Leave a comment

Computing kern.maxfilesperproc and kern.maxfiles

Many thanks to Julien Charbon, a colleague of mine, for providing this information. This is specific to releng/8.3 because we currently utilize this version.

kern.maxfiles is derived from max.maxproc (max number of process) which is derived from max.maxusers (not the maximum number of users, but more a global order of magnitude of machine process/file descriptor usage). max.maxusers is derived from the amount of physical memory and capped up to 384:


        /* Base parameters */
        maxusers = MAXUSERS;
        TUNABLE_INT_FETCH("kern.maxusers", &maxusers);
        if (maxusers == 0) {
                maxusers = physpages / (2 * 1024 * 1024 / PAGE_SIZE);
                if (maxusers < 32)
                        maxusers = 32;
                if (maxusers > 384)
                        maxusers = 384;

A machine with a large amount of memory, 72 GB for example, gets the maximum:

$ sysctl kern.maxusers
kern.maxusers: 384

in sys/kern/subr_param.c:

#define NPROC (20 + 16 * maxusers)
#ifndef NBUF
#define NBUF 0
#ifndef MAXFILES
#define MAXFILES (maxproc * 2)


      * The following can be overridden after boot via sysctl.  Note:
      * unless overriden, these macros are ultimately based on maxusers.
     maxproc = NPROC;
     TUNABLE_INT_FETCH("kern.maxproc", &maxproc);
      * Limit maxproc so that kmap entries cannot be exhausted by
      * processes.
     if (maxproc &gt; (physpages / 12))
             maxproc = physpages / 12;
     maxfiles = MAXFILES;
     TUNABLE_INT_FETCH("kern.maxfiles", &maxfiles);
     maxprocperuid = (maxproc * 9) / 10;
     maxfilesperproc = (maxfiles * 9) / 10;


kern.maxproc = (20 + 16 * maxusers) = (20 + 16 * 384) = 6164
kern.maxfiles = (maxproc * 2) = (2 * 6164) = 12328
kern.maxfilesperproc = (maxfiles * 9) / 10 = (12328 * 9) / 10 = 11095

The math confirmed:

$ sysctl kern.maxproc
kern.maxproc: 6164
$ sysctl kern.maxfiles
kern.maxfiles: 12328
$ sysctl kern.maxfilesperproc
kern.maxfilesperproc: 11095

Thus these low default maximum values are the result of a tentative from FreeBSD to “auto-tune” the default maximum from memory size. However by limiting the kern.maxusers to 384, it concerns only system with less 192 KB of memory (embedded system?).

We located this patch which has the effect of increasing these tunables. See below.

$ sysctl kern.maxfiles
kern.maxfiles: 98312
$ sysctl kern.maxfilesperproc
kern.maxfilesperproc: 88480
Categories: FreeBSD

MeetBSD California 2012

November 7, 2012 1 comment
I am a fairly recent newcomer to the FreeBSD community and had the opportunity to attend my first BSD related conference. This is the account of my experience at MeetBSD California 2012.

MeetBSD California 2012 was coordinated by iX Systems and hosted at Yahoo! corporate headquarters in Sunnyvale, CA. The Yahoo! facility easily accommodated the 100 – 125 attendees in multiple conference rooms for varying purposes across the two days allocated.

Conference Format

The conference was spread across two days. Day 1 revolved around speakers and presentations while day 2 followed what was called “The unConference” agenda. “The unConference” included lightning talks/speed geeking sessions and break out sessions. The topics of the second day were chosen by conference attendees as opposed to the organizers.

The Conference: Day 1



All of the speakers were knowledgeable on their topics and well prepared with slide decks accompanying their presentations some even including demos. Attendees participated with questions, comments, and general discussion. The environment was engaging and lively and provided much information to absorb.

In the large conference room, however, it was difficult for some in the back to hear the speakers. I think an audio sound system would have been an excellent idea for this conference.

The Conference: Day 2

Day 2: The unConference

The idea of the speed geeking session was of particular interest to most in attendance and broke up the monotony of a constant stream of speakers and presentations. This session was a set of speakers covering a topic with a group of individuals for approximately 10 minutes. At the conclusion of that 10 minutes, each group rotated to a different speaker who spoke on their topic for 10 minutes. The rotation was repeated until each group had heard all of the speakers.

Later, Conference attendees voted on subjects that would later become the topics of the breakout sessions. These breakout sessions enabled groups of interested attendees to interactively discuss subjects with which they had definitive interest. These sessions occurred at the latter part of the day. However, as the day wore on, I began to lose focus.

Overall Experience

In conclusion, my experience at the conference was extremely positive. The conference content was very informative and presented well. Additionally, much was gained outside the confines of the conference itself. I had the opportunity to meet and socialize with multiple individuals who are BSD enthusiasts and others with whom I’ve communicated with digitally previously.

FreeBSD kernel panic in udp_input()

November 5, 2012 1 comment

FreeBSD kernel panic in udp_input()

The Problem

Running FreeBSD 8.3, this kernel panic was produced under UDP load:

Fatal trap 12: page fault while in kernel mode
cpuid = 4; apic id = 08
fault virtual address = 0x0
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff807a08a2
stack pointer = 0x28:0xffffffa40c2689b0
frame pointer = 0x28:0xffffffa40c268a80
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 12 (swi1: netisr 0)
trap number = 12
panic: page fault
cpuid = 4
KDB: stack backtrace:
#0 0xffffffff80642b3e at kdb_backtrace+0x5e
#1 0xffffffff8060fd57 at panic+0x187
#2 0xffffffff80905990 at trap_fatal+0x290
#3 0xffffffff80905ce1 at trap_pfault+0x201
kernel trap 12 with interrupts disabled
#4 0xffffffff8090619f at trap+0x3df

#5 0xffffffff808ed674 at calltrap+0x8

#6 0xffffffff8072000c at ip_input+0xac
Fatal trap 12: page fault while in kernel mode
cpuid = 5; apic id = 0a
#7 0xffffffff806cba29 at swi_net+0x149
fault virtual address = 0xc
#8 0xffffffff805e7794 at intr_event_execute_handlers+0x104
fault code = supervisor read data, page not present
#9 0xffffffff805e8e25 at ithread_loop+0x95
instruction pointer = 0x20:0xffffffff8064f795
#10 0xffffffff805e49af at fork_exit+0x11f
stack pointer = 0x28:0xffffffa41cd969a0
#11 0xffffffff808edbbe at fork_trampoline+0xe
frame pointer = 0x28:0xffffffa41cd969d0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = resume, IOPL = 0
current process = 8 (pagedaemon)
trap number = 12

Marc de la Gerroniere and Julien Charbon, developer colleagues of mine, commenced an intense troubleshooting exercise that lasted several days. Thanks to them for providing the patch below.

The Issue

The kernel panic occurred when a process executes `sysctl net.inet.udp.pcblist` followed by a UDP socket being closed and then receiving a packet on that socket before the process executing sysctl had completed.

The Patch

Applying this patch to FreeBSD sources and building resolves the issue above.

commit 497aa2bc6f4b7383344ab7fd812d296c6997298d
Author: Marc de la Gueronniere 
Date:   Wed Oct 24 19:59:41 2012 +0000

    kern/172963: Kernel panic in udp_input()

diff --git a/sys/netinet/udp_usrreq.c b/sys/netinet/udp_usrreq.c
index b720364..f9d0245 100644
--- a/sys/netinet/udp_usrreq.c
+++ b/sys/netinet/udp_usrreq.c
@@ -494,6 +494,12 @@ udp_input(struct mbuf *m, int off)
+                     // in_pcbdrop does not unlink from V_udb
+                      if (inp->inp_socket == NULL) {
+                             INP_RUNLOCK(inp);
+                             continue;
+                     }
                      * Handle socket delivery policy for any-source
                         * and source-specific multicast. [RFC3678]
@@ -1559,6 +1565,7 @@ udp_detach(struct socket *so)
        KASSERT(up != NULL, ("%s: up == NULL", __func__));
   inp->inp_ppcb = NULL;
+    in_pcbdrop(inp);
diff --git a/sys/netinet6/udp6_usrreq.c b/sys/netinet6/udp6_usrreq.c
index 22ddde4..2e8b564 100644
--- a/sys/netinet6/udp6_usrreq.c
+++ b/sys/netinet6/udp6_usrreq.c
@@ -271,6 +271,10 @@ udp6_input(struct mbuf **mp, int *offp, int proto)
                                    inp->inp_fport != uh->uh_sport)
+                     // in_pcbdrop does not unlink from V_udb
+                      if (inp->inp_socket == NULL) {
+                             continue;
+                     }
                      * Handle socket delivery policy for any-source
@@ -964,6 +968,7 @@ udp6_detach(struct socket *so)
     up = intoudpcb(inp);
   KASSERT(up != NULL, ("%s: up == NULL", __func__));
+    in_pcbdrop(inp);


A “Problem Report” (aka PR) was submitted to the FreeBSD project; it can be viewed at

Categories: FreeBSD

So You Want A New FreeBSD Boot Loader Option…

October 11, 2012 2 comments

Creating A New FreeBSD Boot Loader Option

The Use Case

There are multiple kernels installed on the FreeBSD host and an option to boot to this kernel is necessary.


The Patch

This is the patch. Copy/paste it to /boot/beastie.patch for use in the procedure below.

Index: beastie.4th
--- beastie.4th
+++ beastie.4th
@@ -41,6 +41,7 @@
 variable bootkey
 variable bootacpikey
 variable bootsafekey
+variable bootdebugkey
 variable bootverbosekey
 variable bootsinglekey
 variable escapekey
@@ -194,6 +195,7 @@
 	printmenuitem ."  Boot FreeBSD in Safe Mode" bootsafekey !
 	printmenuitem ."  Boot FreeBSD in single user mode" bootsinglekey !
 	printmenuitem ."  Boot FreeBSD with verbose logging" bootverbosekey !
+	printmenuitem ."  Boot FreeBSD with debug kernel" bootdebugkey !
 	printmenuitem ."  Escape to loader prompt" escapekey !
 	printmenuitem ."  Reboot" rebootkey !
 	menuX @ 20 at-xy
@@ -286,6 +288,9 @@
 			s" YES" s" boot_single" setenv
 			0 boot
+		dup bootdebugkey @ = if
+			s" /boot/DEBUG/kernel" 1 boot
+		then
 		dup escapekey @ = if
 			s" NO" s" autoboot_delay" setenv

Applying The Patch

Follow these steps to apply the patch to your target hosts:

# cd /boot
# cp beastie.4th beastie.4th.orig
# patch -p0 < beastie.patch

The new option will be loaded when loader called beastie.4th.

Categories: FreeBSD

Building FreeBSD Media With Custom Packages

October 8, 2012 Leave a comment

Building FreeBSD Media With Custom Packages

Building a FreeBSD release can be as simple as executing:

# cd /usr/src
# make buildworld
# cd release
# make release
# PKG_TREE=$pkg_dir PKG_DEST=$CHROOTDIR/R/cdrom/dvd1 PKG_DVD=Yes make package-split
This post expands on the `make package-split` command whose purpose is to create a packages directory.

The Use Case

I generate company specific FreeBSD distributions based on our internal requirements including loading specific packages. The FreeBSD DVD release from includes a packages/ directory created by the script. I use the same script to accomplish the same task with custom packages.

We generate an ISO from the release and import it into Cobbler. We’ve setup Cobbler to allow us to provision operating systems non-interactively using PXE, DHCP, TFTP, and HTTP.


  • A directory containing at least the packages to include. One can download packages for 8.3-RELEASE via ftp at
  • An INDEX file. The INDEX file will already exist if one downloads the packages from the above ftp server.

Configuring is a Python script that “generates a master INDEX file for the CD images built by the FreeBSD release engineers. Each disc is given a list of desired packages. Dependencies of these packages are placed on either the same disc or an earlier disc. The resulting master INDEX file is then written out”.

In our use case, the files are not split into separate discs because we are packaging for a DVD as identified by the PKG_DVD environment variable.

View the here. Because I am currently working with 8.3-RELEASE, I link directly to the that version of the script.

Our use case involves commenting lines 43 – 81 and copying lines 63 – 81. The copied lines are modified with names of packages installed in our base distribution. An example might look similar to:

if doing_dvd:


  1. There are inconsistencies in the INDEX file where some package names and origins do not match. An example such as ruby18-iconv shows the package name (column 1) is ruby18-iconv-$version while the origin (column 2) is /usr/ports/converters/ruby-iconv. `make package-split` errored on this package citing “Unable to find package for converters/ruby18-iconv” (see image below). Changing origin from “ruby-iconv” to “ruby18-iconv” resolved this.
  2. use the origin field to identify the package in configuring the script. For example, ruby18-iconv, would be specified as ‘converters/ruby19-iconv’ as opposed to ‘ruby/ruby18-iconv’ despite having multiple categories and existing multiple directories.

Executing `make package-split`

Once the release is built, create the packages directory and move it to the new release by executing the following. This would also be an appropriate time to make additional modifications to the image such as replacing the mfsroot.gz with a new custom mfsroot.

# PKG_TREE=/tmp/packages PKG_DEST=$CHROOTDIR/R/cdrom PKG_DVD=Yes make package-split
# mv $CHROOTDIR/R/cdrom/disc1/packages $CHROOTDIR/R/cdrom/dvd1

Generate the ISO

The release is now ready to be made into an ISO that we use to import the distribution into our provisioning platform, Cobbler.

# mkisofs -R -no-emul-boot -b boot/cdboot -o /tmp/iso/FreeBSD-8.3-RELEASE-p4-amd64-CUSTOM.iso $CHROOTDIR/R/cdrom/dvd1
Categories: FreeBSD

Building FreeBSD With Multiple Kernels

October 1, 2012 Leave a comment

Building a Release With Multiple Kernels

The Use Case

FreeBSD installs a run-time kernel used during normal day-to-day operations. In the event of repeated kernel panics, there may be a need for a kernel capable of online debugging.

Patching sysinstall

sysinstall(8) lacks support for installing multiple kernel distributions in releases 8.3 or earlier. Patch r240972 was committed to stable/8 enabling support for a DEBUG kernel distribution. The procedure below patches the sources with bits necessary for the remainder of this post.

# cd /usr/src
# svn diff -c 240972 > sysinstall_patch
# patch -p0 < sysinstall_patch

Replacing the string DEBUG in the patch changes the kernel distribution name.

Preparing The Build

The only prerequisite is having FreeBSD sources available and, optionally, ports. Ports is obtained via cvs or Subversion. Information on FreeBSD source trees are found here.

We assume a FreeBSD release build with two kernels, GENERIC and DEBUG, on the amd64 architecture. To avoid using CVS/SVN as part of the build process, it is helpful to have a local copy of ports and FreeBSD sources. In this example, ports is /usr/ports and the sources are /usr/src.

First we must buildworld:

# cd /usr/src && make buildworld

The Kernel Config

The FreeBSD release build process builds the GENERIC kernel by default. Adding an additional kernel is accomplished by first creating the kernel config file. We will create a kernel config called /usr/src/sys/amd64/conf/DEBUG. The kernel config should appear as follows:

include GENERIC
options DDB

Compiling The Release

Now compile the release (as root, in bash):

# cd release

This example builds a DVD image, no floppy images, no documentation, with ports and sources at /usr/ports and /usr/src, respectively. The DEBUG kernel and config file are defined by the KERNELS, KERNCONF, and INSTKERNNAME environment variables. Not all of these are required and operators may choose options not shown here. Information on available environment variables are found in the release(7) manpage. Many of these options can be set in /usr/src/release/Makefile.

The CVSROOT environment variable is non-optional. It is used to download ports and sources. If the environment variables EXTPORTSDIR and EXTSRCDIR are set, `make release` will not consult CVSROOT, despite being non-optional.

Executing/Using The Kernel

From the loader menu

Press 6 at the loader menu to escape to the loader(8) prompt and type:

boot DEBUG

From the running system

The DEBUG kernel is loaded by default when the kernel line in loader.conf appears as follows. Each subsequent boot loads the DEBUG kernel.


Dropping into the debugger

A kernel panic induces a system reboot unless debug.debugger_on_panic is enabled as follows:

sysctl debug.debugger_on_panic=1

To induce a panic:

sysctl debug.kdb.panic=1

Categories: FreeBSD

So You Want A FreeBSD Debugging Kernel…

September 25, 2012 1 comment

Installing a debugging kernel in FreeBSD

The Use Case

There are numerous reasons for needing a debugging kernel. For example, during a kernel panic. This was the case with us. Therefore, we installed a kernel with DDB enabled. In the event of a kernel panic, the system would drop to the debugger and the software engineers were then able to begin diagnosing the problem.

Preparing To Build The Kernel


FreeBSD Sources

Operators can use cvs or Subversion to install the sources. Information on FreeBSD source trees can be found here.

The Kernel Config

Create a kernel config by opening /usr/src/sys/${arch}/conf/DEBUG. Where ${arch} is the machine architecture the kernel is to be built for. In this case, we build an amd64 kernel on an amd64 machine. The kernel config should appear as follows:

include GENERIC
options DDB

Compiling and Installing The Kernel

Compiling and installing the debugging kernel is as simple as executing:

# cd /usr/src

Executing/Using The Kernel

From the loader menu

At the loader menu, press 6 to escape to the loader(8) prompt and type:

boot DEBUG

From the running system

The DEBUG kernel can be loaded by default by adding/modifying the kernel directive in loader.conf as follows. Each subsequent boot will load the DEBUG kernel.


Dropping into the debugger

During a kernel panic, it will simply reboot unless debug.debugger_on_panic is enabled. To enable this execute:

sysctl debug.debugger_on_panic=1

To induce a panic:

sysctl debug.kdb.panic=1

Categories: FreeBSD