The Problem

Running FreeBSD 8.3, this kernel panic was produced under UDP load:

Fatal trap 12: page fault while in kernel mode
cpuid = 4; apic id = 08
fault virtual address = 0x0
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff807a08a2
stack pointer = 0x28:0xffffffa40c2689b0
frame pointer = 0x28:0xffffffa40c268a80
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 12 (swi1: netisr 0)
trap number = 12
panic: page fault
cpuid = 4
KDB: stack backtrace:
#0 0xffffffff80642b3e at kdb_backtrace+0x5e
#1 0xffffffff8060fd57 at panic+0x187
#2 0xffffffff80905990 at trap_fatal+0x290
#3 0xffffffff80905ce1 at trap_pfault+0x201
kernel trap 12 with interrupts disabled
#4 0xffffffff8090619f at trap+0x3df

#5 0xffffffff808ed674 at calltrap+0x8

#6 0xffffffff8072000c at ip_input+0xac
Fatal trap 12: page fault while in kernel mode
cpuid = 5; apic id = 0a
#7 0xffffffff806cba29 at swi_net+0x149
fault virtual address = 0xc
#8 0xffffffff805e7794 at intr_event_execute_handlers+0x104
fault code = supervisor read data, page not present
#9 0xffffffff805e8e25 at ithread_loop+0x95
instruction pointer = 0x20:0xffffffff8064f795
#10 0xffffffff805e49af at fork_exit+0x11f
stack pointer = 0x28:0xffffffa41cd969a0
#11 0xffffffff808edbbe at fork_trampoline+0xe
frame pointer = 0x28:0xffffffa41cd969d0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = resume, IOPL = 0
current process = 8 (pagedaemon)
trap number = 12

Marc de la Gerroniere and Julien Charbon, developer colleagues of mine, commenced an intense troubleshooting exercise that lasted several days. Thanks to them for providing the patch below.

The Issue

The kernel panic occurred when a process executes `sysctl net.inet.udp.pcblist` followed by a UDP socket being closed and then receiving a packet on that socket before the process executing sysctl had completed.

The Patch

Applying this patch to FreeBSD sources and building resolves the issue above.

commit 497aa2bc6f4b7383344ab7fd812d296c6997298d
Author: Marc de la Gueronniere 
Date:   Wed Oct 24 19:59:41 2012 +0000

    kern/172963: Kernel panic in udp_input()

diff --git a/sys/netinet/udp_usrreq.c b/sys/netinet/udp_usrreq.c
index b720364..f9d0245 100644
--- a/sys/netinet/udp_usrreq.c
+++ b/sys/netinet/udp_usrreq.c
@@ -494,6 +494,12 @@ udp_input(struct mbuf *m, int off)
 
                       INP_RLOCK(inp);
 
+                     // in_pcbdrop does not unlink from V_udb
+                      if (inp->inp_socket == NULL) {
+                             INP_RUNLOCK(inp);
+                             continue;
+                     }
+
                    /*
                      * Handle socket delivery policy for any-source
                         * and source-specific multicast. [RFC3678]
@@ -1559,6 +1565,7 @@ udp_detach(struct socket *so)
        KASSERT(up != NULL, ("%s: up == NULL", __func__));
   inp->inp_ppcb = NULL;
       in_pcbdetach(inp);
+    in_pcbdrop(inp);
       in_pcbfree(inp);
       INP_INFO_WUNLOCK(&V_udbinfo);
      udp_discardcb(up);
diff --git a/sys/netinet6/udp6_usrreq.c b/sys/netinet6/udp6_usrreq.c
index 22ddde4..2e8b564 100644
--- a/sys/netinet6/udp6_usrreq.c
+++ b/sys/netinet6/udp6_usrreq.c
@@ -271,6 +271,10 @@ udp6_input(struct mbuf **mp, int *offp, int proto)
                                    inp->inp_fport != uh->uh_sport)
                                      continue;
                      }
+                     // in_pcbdrop does not unlink from V_udb
+                      if (inp->inp_socket == NULL) {
+                             continue;
+                     }
 
                    /*
                      * Handle socket delivery policy for any-source
@@ -964,6 +968,7 @@ udp6_detach(struct socket *so)
     up = intoudpcb(inp);
   KASSERT(up != NULL, ("%s: up == NULL", __func__));
   in_pcbdetach(inp);
+    in_pcbdrop(inp);
       in_pcbfree(inp);
       INP_INFO_WUNLOCK(&V_udbinfo);
      udp_discardcb(up);

References

A “Problem Report” (aka PR) was submitted to the FreeBSD project; it can be viewed at http://www.freebsd.org/cgi/query-pr.cgi?pr=172963.