Manual Package Builds For FreeBSD < 10.0-RELEASE

May 28, 2014 1 comment

Just wanted to link to an older blog post I found useful recently…

Given the current state of migration from legacy pkg_* tools to pkg(8) within FreeBSD, the content here is old, but still useful for those that haven’t migrated to 10.x (or newer) or the new pkg(8) framework.

Check out Troy’s Unix Space post on package builds on blogspot.

Disclaimer

Data and information described on this blog are for informational purposes only. The author of this blog provides no warranty/guarantee, expressed or implied, that this data and information will function as described here. Readers are expected to exercise due diligence when researching, developing, and deploying techniques and methods for use within their environments.

Comments posted are the explicit opinions of the comment poster themselves and does not necessarily reflect the views and opinions of the author of this blog.

Advertisements
Categories: FreeBSD

BSDCan 2014 Travel Woes

May 16, 2014 3 comments
I wasn’t going to post anything about my travel woes to Ottawa, ON for BSDCan 2014, but upon receiving the revised itinerary after the 3rd flight cancellation, I decided it was too humorous to pass up…

Originally scheduled to fly out on Wednesday via United, I arrived at the airport, parked my vehicle, removed my bags, and proceeded to walk to the main terminal.  My phone vibrated while walking to United’s check-in counters…I just received a flight cancellation notice citing “severe weather conditions in our route network” despite there being little weather in the areas of Dulles and Ottawa that day.  A United representative rescheduled the flight the following morning which was subsequently cancelled for the same reason…still no real weather to be seen, but there was certainly weather on the way for Thursday evening into Friday.

I contacted the travel agent following the 2nd cancellation and discussed two options;  A connecting flight with a 4 hour layover that day which ended up being cancelled or a non-stop flight Friday morning.  Seeing as how the total time on the former flight would have been 8 – 10 hours, I opted for the latter, shorter, 1.5 hour flight Friday morning, just in time to attend most of BSDCan’s main conference activities.

I awoke Friday morning to a deluge of rain thinking the flight would certainly be cancelled, but the flight status, according to United, was “on schedule”.  I got up and continued my daily morning routine intending to make the drive to the airport while checking the phone periodically for flight status updates.  The phone vibrated as I was preparing to leave home with an email update that the flight was cancelled.  It was the cancellation I had been expecting so, instead of driving to the airport, I continued into the office.

Shortly after arriving at the office, I received an email from United with the revised itinerary:

The irony in the revised itinerary is two-fold in that I depart Dulles to Ottawa on the same plane I was originally scheduled to return to Dulles from Ottawa on and that United thinks it’s ok for me to fly from Dulles to Ottawa, stay on the plane, and fly back to Dulles approximately 30 minutes later.

Categories: Conferences & Summits

FreeBSD 10.0: release.sh mapped

February 24, 2014 1 comment

FreeBSD‘s release.sh is a shell script introduced in FreeBSD 9.x whose purpose is to automate FreeBSD release building from source.  This post maps the release.sh into a table of variables and a flowchart describing the program flow and is based on release.sh as it exists in releng/10.0 on FreeBSD‘s subversion or GitHub.

Variables

The following table describes each available variable. The “Default” column contains strings which each variable is set to, with the exception of “Unset” and “NULL”. Unset variables are uninitialized while NULL variables are initialized with no value. Required variables are denoted with an asterisk (*) immediately appended to the name.

Variable Purpose Default
PATH* The environment’s $PATH /bin:/sbin:/usr/bin:
/usr/sbin:/usr/local/bin
CHROOTDIR* The relative or fully qualified path of the chroot where non-contaminated distributions are built /scratch
SVNROOT* The protocol and address of the subversion repository from which sources, docs, and ports are checked out from svn://svn.freebsd.org
SRCBRANCH* The source branch to checkout, compile, and install base/head@rHEAD
DOCBRANCH The doc branch to checkout, compile, and install docs/head@rHEAD
PORTBRANCH The ports branch to checkout, compile, and install ports/head@rHEAD
SRC_FORCE_CHECKOUT Enables use of “–force” with the svn command line NULL
MAKE_CONF* The path to the make.conf file to implement in the chroot for release builds /dev/null
SRC_CONF* The path to the src.conf file to implement in the chroot for release builds /dev/null
NCPU* Derives the number of CPU cores available
WORLD_FLAGS make buildworld make flags. Sets -j to $NCPU Unset
KERNEL_FLAGS make buildkernel make flags. Sets -j to $NCPU / 2 Unset
MAKE_FLAGS* Set’s the -s flag preventing the commands from being echoed to STDOUT -s
KERNEL* Specify the kernel to build with the distribution GENERIC
WITH_DVD Enables the generation of a DVD ISO of the resulting image NULL
RELEASECONF Enables operators/engineers to specify a custom release.conf on the command line Unset
NOPORTS If set, prevents the inclusion of a ports tree in the resulting distribution NULL
NODOC If set, prevents the inclusion of a docs tree in the resulting distribution NULL
DOCPORTS If NOPORTS and/or NODOC are unset, they must not pass to make as variables. The release Makefile verifies definedness of NOPORTS/NODOC variables instead of their values. NULL
CONF_FILES* The aggregated build-time flags based upon variables defined within this file, unless overridden by release.conf. In most cases, these will not need to be changed. __MAKE_CONF=${MAKE_CONF} SRCCONF=${SRC_CONF}
__MAKE_CONF Permits the specification of an alternate make.conf Unset
SRCCONF Equal to SRC_CONF /dev/null
TARGET A make variable for cross-compiling releases
TARGET_ARCH A make variable for cross-compiling releases
ARCH_FLAGS Variable specifying architecture dependent built options
CHROOT_WMAKEFLAGS* make buildworld flags for the chroot environment ${MAKE_FLAGS} ${WORLD_FLAGS} ${CONF_FILES}
CHROOT_IMAKEFLAGS* make installworld flags for the chroot environment ${CONF_FILES}
CHROOT_DMAKEFLAGS* make distribution flags for the chroot environment ${CONF_FILES}
RELEASE_WMAKEFLAGS* make buildworld flags for the release inside the chroott ${MAKE_FLAGS} ${WORLD_FLAGS} ${ARCH_FLAGS} ${CONF_FILES}
RELEASE_KMAKEFLAGS* make buildkernel flags for the release inside the chroot ${MAKE_FLAGS} ${KERNEL_FLAGS} KERNCONF=${KERNEL} ${ARCH_FILES} ${CONF_FILES}
RELEASE_RMAKEFLAGS* make release build flags inside the chroot ${ARCH_FILES} KERNCONF=${KERNEL} ${CONF_FILES} ${DOCPORTS}
WITH_DVD=${WITH_DVD}
FORCE_SRC_KEY Enables use of “–force” with the svn command line NULL
RELSTRING* The release string as returned from uname -s Unset
PBUILD_FLAGS Specifies port build flags for docproj Unset
_OSVERSION
OSVERSION
Identifies the OS release date and is included in PBUILD_FLAGS for the docproj port Unset

Flowchart

The following chart describes the release.sh program flow:

Disclaimer

Data and information described on this blog are for informational purposes only. The author of this blog provides no warranty/guarantee, expressed or implied, that this data and information will function as described here. Readers are expected to exercise due diligence when researching, developing, and deploying techniques and methods for use within their environments.

Comments posted are the explicit opinions of the comment poster themselves and does not necessarily reflect the views and opinions of the author of this blog.

Categories: FreeBSD

FreeBSD Ports: Managing Custom Ports/Physical Categories

December 16, 2013 3 comments

Managing Custom Ports/Physical Categories

Overview

This post describes a method of managing custom FreeBSD Ports involving creating a directory (a.k.a. a physical category) of subdirectories; Each an individual port containing necessary files, e.g. Makefile, etc.

I found this process, documented in the FreeBSD Committer’s Guide, to be confusing as the answer provided does not directly address the question. It describes moving a port as opposed to creating a new category/port, but it does encompass this process. Therefore, the answer is unclear and contains superfluous information.

We simplify the instructions for creating physical categories to augment the instructions for moving a Port. I assume the reader has previously prepared a port which exists on disk. Creating a port and working with version control systems is beyond the scope of this post.

Please bear in mind, My Git-fu is rudimentary at best. If others, more knowledgeable in this area, can comment on other ways to accomplish the same, please do.

The Use Case

Engineers and administrators deploying custom software may wish to build and distribute packages for these applications via FreeBSD’s Ports system. There are two options for using Ports; We will be addressing the latter in this post:

  • Submit to the official FreeBSD Ports collection
  • Create an internal custom Ports collection

The Procedure(s)

We start by cloning FreeBSD Ports from GitHub and pulling a customized Ports collection (maintained internally) into our local FreeBSD Ports repository. We’ll finish by describing the process for pulling updates into the custom tree from each of the two repos.

Creating The Repo

  1. git clone https://www.github.com/freebsd/freebsd-ports.git
  2. cd freebsd-ports
  3. git remote add repo /path/to/local/repo
  4. git pull repo master
  5. cd $new_physical_category/$custom_port && make describe (optional, reports errors if they exist)
  6. run portlint on the port (Verifies correct state of Port)
  7. env PORTSDIR=/path/to/ports sh -e /path/to/ports/Tools/scripts/chkorigin.sh
  8. cd /path/to/combined/ports_tree && make checksubdirs && make index

Updating the Repo

  1. cd /path/to/local/freebsd-ports/clone
  2. git pull origin
  3. git pull repo master

If new physical categories and/or ports are added to the internal repo, steps 5 – 8 above need to be executed. Other changes may or may not require this.

Summary

This procedure creates a combined Ports collection from two sources, FreeBSD’s official Ports collection and a private collection. Once completed, normal operations can be performed on the resulting Ports tree as necessary such as installing applications or creating packages, etc.

One use of this repo is to commit to git and tag each commit as a new version of the custom Ports collection thus making them easily recallable.

Disclaimer

Data and information described on this blog are for informational purposes only. The author of this blog provides no warranty/guarantee, expressed or implied, that this data and information will function as described here. Readers are expected to exercise due diligence when researching, developing, and deploying techniques and methods for use within their environments.

Comments posted are the explicit opinions of the comment poster themselves and does not necessarily reflect the views and opinions of the author of this blog.

Categories: FreeBSD

vBSDcon Wrap-Ups

November 20, 2013 1 comment

vBSDcon Wrap-Ups!

Verisign’s first-ever vBSDcon was a very successful event which was held October 25 – 27, 2013 at the Dulles Hyatt in Herndon, VA.  Speakers and attendees from all over the US and Europe convened at vBSDcon for exclusive networking opportunities, plenary talks, birds of a feathers, and speed geeking.

In the weeks following the conference, several organizations posted follow-ups of the event.  We’d like to encourage you to take a few minutes to read through them at the links below.

  • iXsystems, a vBSDcon Gold sponsor, posted this write-up
  • RootBSD, the vBSDcon Tote sponsor, posted this write-up
  • BSDNow, a weekly BSD news podcast, has a podcast review of vBSDcon here
  • Undeadly.org published a small article based on Henning Brauer and Reyk Floeter’s experiences here.
  • Check out the million+ impression tweets here
  • John-Mark Gurney provided a trip report here
  • Verisign’s own wrap-up, by CTO Burt Kaliski, is here

OpenBSD at vBSDcon October 25 – 27, 2013 in Virginia

October 8, 2013 Leave a comment

OpenBSD at vBSDcon October 25 – 27, 2013 in Virginia

For only USD$75 you can register for vBSDcon hosted by Verisign on October 25 – 27, 2013 in Herndon, VA. That is less than 3 weeks away! If you have not registered yet, it is definitely recommended as vBSDcon will feature a series of roundtable discussions, educational sessions, best practice conversations, and exclusive networking opportunities. Registrations for vBSDcon will be open until October 23, 2013 at http://www.vbsdcon.com/.

vBSDcon will feature developers, Henning Brauer and Reyk Floeter, from the OpenBSD project. Henning and Reyk will be tag teaming a presentation on packet inspection with pf in OpenBSD. pf was introduced in 2001 as an alternative to ipf. Over the years, pf has become a mature, secure, and powerful, yet easy to use high performance packet filter. In this presentation, Henning Brauer will describe some of the features of pf and Reyk Floeter will describe utilizing relayd in conjunction with pf. You can expect to hear about using hooks for transparent proxies, deep packet inspection, socket splicing, NATs, load balancing and more.

Read more about our speakers and their topics, the conference agenda, other activities, and registrations at http://www.vbsdcon.com/. This is an event you will not want to miss. Register now before it’s too late!

Follow @VERISIGN and @hostileaddmin on Twitter for more news and updates on #vBSDcon

Enabling mrepo Support for FreeBSD Packages

October 2, 2013 Leave a comment

Enabling mrepo Support for FreeBSD Packages

mrepo is a content management application for Yum/APT RPM Repositories which does not support FreeBSD packages. While this is not necessarily an elegant solution, it’s enough to enable support for FreeBSD packages and can get one started in deploying a more proper implementation.

Enabling preliminary support for FreeBSD packages involves the following procedure. This procedure assumes repos are available via HTTP at the location where content is downloaded from. We also assume a web server is installed and configured to serve mrepo content. Any discussion involving the configuration of these components is beyond the scope of this post.

  1. Apply the patch below to the mrepo.conf and add some FreeBSD repos
  2. Update the local mrepo repositories
  3. Execute `find` symlinking to the updated content
  4. Setup cronjobs to update repos periodically

Applying The Patch

This patch was generated with GNU patch on RHEL6 and should be applied using GNU patch in the following manner to minimize potential issues. It patches the mrepo script, mrepo.conf, and adds three mrepo repositories. The patch:

diff -u usr/bin/mrepo usr/bin/mrepo.new
--- usr/bin/mrepo	2011-07-01 21:16:09.000000000 -0400
+++ usr/bin/mrepo.new	2013-09-26 12:10:22.072081702 -0400
@@ -214,6 +214,7 @@
         self.lftpcleanup = self.getoption('main', 'lftp-cleanup', 'yes') not in disable
         self.lftpexcldebug = self.getoption('main', 'lftp-exclude-debug', 'yes') not in disable
         self.lftpexclsrpm = self.getoption('main', 'lftp-exclude-srpm', 'yes') not in disable
+        self.lftpincltbz = self.getoption('main', 'lftp-include-tbz', 'yes') not in disable
         self.lftpoptions = self.getoption('main', 'lftp-options', '')
         self.lftpcommands = self.getoption('main', 'lftp-commands', '')
         self.lftpmirroroptions = self.getoption('main', 'lftp-mirror-options', '-c -P')
@@ -1282,6 +1283,8 @@
         mirroropts = mirroropts + ' -X \"*.src.rpm\" -X \"/SRPMS/\"'
     if cf.lftpexcldebug:
         mirroropts = mirroropts + ' -X \"*-debuginfo-*.rpm\" -X \"/debug/\"'
+    if cf.lftpincltbz:
+        mirroropts = mirroropts + ' -I \"*.tbz*\"'
 
     ret = run('%s %s -c \'%s mirror %s %s %s\'' % (cf.cmd['lftp'], opts, cmds, mirroropts, url, path),dryrun=True)
     if ret:
diff -ruN etc/mrepo.conf newetc/mrepo.conf
--- etc/mrepo.conf	2013-09-30 10:57:32.036840555 -0400
+++ newetc/mrepo.conf	2013-09-30 09:08:34.960901787 -0400
@@ -1,4 +1,4 @@
-
+### Configuration file for mrepo
 
 ### The [main] section allows to override mrepo's default settings
 ### The mrepo-example.conf gives an overview of all the possible settings
@@ -23,3 +23,5 @@
 hardlink = no
 shareiso = no
 rhnget-download-all = yes
+
+lftp-include-tbz = yes
diff -ruN etc/mrepo.conf.d/FreeBSD-8-x86_64-dev.conf newetc/mrepo.conf.d/FreeBSD-8-x86_64-dev.conf
--- etc/mrepo.conf.d/FreeBSD-8-x86_64-dev.conf	1969-12-31 19:00:00.000000000 -0500
+++ newetc/mrepo.conf.d/FreeBSD-8-x86_64-dev.conf	2013-09-30 10:59:07.834704096 -0400
@@ -0,0 +1,5 @@
+[FreeBSD-8-x86_64-dev]
+name = FreeBSD 8.x Packages - Dev Repo
+release = 8
+metadata = yum repomd
+dev-repo = $url_to_the_development_repo
diff -ruN etc/mrepo.conf.d/FreeBSD-8-x86_64-prod.conf newetc/mrepo.conf.d/FreeBSD-8-x86_64-prod.conf
--- etc/mrepo.conf.d/FreeBSD-8-x86_64-prod.conf	1969-12-31 19:00:00.000000000 -0500
+++ newetc/mrepo.conf.d/FreeBSD-8-x86_64-prod.conf	2013-09-30 10:59:18.794803601 -0400
@@ -0,0 +1,5 @@
+[FreeBSD-8-x86_64-prod]
+name = FreeBSD 8.x Packages - Production Repo
+release = 8
+metadata = yum repomd
+prod-repo = $url_to_the_production_repo
diff -ruN etc/mrepo.conf.d/FreeBSD-8-x86_64-qa.conf newetc/mrepo.conf.d/FreeBSD-8-x86_64-qa.conf
--- etc/mrepo.conf.d/FreeBSD-8-x86_64-qa.conf	1969-12-31 19:00:00.000000000 -0500
+++ newetc/mrepo.conf.d/FreeBSD-8-x86_64-qa.conf	2013-09-30 10:59:28.171888437 -0400
@@ -0,0 +1,5 @@
+[FreeBSD-8-x86_64-qa]
+name = FreeBSD 8.x Packages - QA Repo
+release = 8
+metadata = yum repomd
+qa-repo = $url_to_the_qa_repo

are applied by executing the following respectively for each patch after the patches have been downloaded:

# cd / && patch < mrepo.patch

Note this patch includes variables that should be replaced by URLs where the repos exist. This URL could be the URL to FreeBSD’s binary package repos, but the size of the repo could cause some issues with this process, especially the step below where the symlinks are generated. My use case involves HTTP repos of home grown FreeBSD packages.

This patch can also be modified to enable support for other file types. For example, tgz and txz files which are supported in FreeBSD’s new pkg framework. which will be default in FreeBSD 10.0-RELEASE. Doing so would enable support for FreeBSD 10 ‘pkg’ files to mrepo as well.

Updating The Repositories

mrepo repositories are updated by executing the patched mrepo script as described here:

# mrepo -ug dev-repo
# mrepo -ug qa-repo
# mrepo -ug prod-repo

These commands update each of the above repos [ included in the patch ] according to srcdir and wwwdir specified in mrepo.conf. Therefore, content downloads to srcdir and wwwdir will be updated in the next step.

Symlink Content

mrepo downloads content from the URLs specified in the mrepo config files located in /etc/mrepo.conf.d/. The FreeBSD-8-x86_64-*.conf repositories in this case. The files are symlinked to from the repo identified by the combination of the value in wwwdir specified in /etc/mrepo.conf and the repo name. This is done by executing the following:

# cd $wwwdir/$repodir && find $srcdir/$repo –type f –name '*tbz' -exec ln –s {} . \;

Where $wwwdir and $srcdir are taken from /etc/mrepo.conf and $repodir is actual directory name of the repo on disk. This step should be executed for each of the repos. Completion of this step means the repos are now available via HTTP.

Configure Cronjobs

It is generally considered a good idea to enable periodic updates of mrepo repos. This can be done in any number of ways, including cronjobs. I choose cronjobs here for simplicity’s sake. Your cronjobs may look similar to the following:

0 0 1 * * mrepo -ug $repo && cd $wwwdir/$repo && find $srcdir/$repo –type f –name ‘*tbz’ -exec ln –s {} . \;

Where $repo is the repo tag specified in it’s config file in /etc/mrepo.conf.d/, $srcdir and $wwwdir are as specified in /etc/mrepo.conf.

Make Content Available Via Cobbler

Some technologists may employ provisioning platforms to distribute OS and package repo contents throughout an environment. I utilize Cobbler in my environments. Therefore, an additional one-time step to create the Cobbler repo is necessary for each repo.

Once the symlinks are created in $wwwdir/$repo, one can add the repo to Cobbler making it available there. This is done by executing:

# cobbler repo add --name="$repo_name" --breed="rsync" --mirror="$wwwdir/$repo"

The new repos will be updated any time a `cobbler reposync` is executed.

Disclaimer

This blog is posted for informational purposes only. Extensive testing is recommended prior to implementing changes discussed here.