Archive

Archive for the ‘FreeBSD’ Category

FreeBSD Ports: Managing Custom Ports/Physical Categories

December 16, 2013 3 comments

Managing Custom Ports/Physical Categories

Overview

This post describes a method of managing custom FreeBSD Ports involving creating a directory (a.k.a. a physical category) of subdirectories; Each an individual port containing necessary files, e.g. Makefile, etc.

I found this process, documented in the FreeBSD Committer’s Guide, to be confusing as the answer provided does not directly address the question. It describes moving a port as opposed to creating a new category/port, but it does encompass this process. Therefore, the answer is unclear and contains superfluous information.

We simplify the instructions for creating physical categories to augment the instructions for moving a Port. I assume the reader has previously prepared a port which exists on disk. Creating a port and working with version control systems is beyond the scope of this post.

Please bear in mind, My Git-fu is rudimentary at best. If others, more knowledgeable in this area, can comment on other ways to accomplish the same, please do.

The Use Case

Engineers and administrators deploying custom software may wish to build and distribute packages for these applications via FreeBSD’s Ports system. There are two options for using Ports; We will be addressing the latter in this post:

  • Submit to the official FreeBSD Ports collection
  • Create an internal custom Ports collection

The Procedure(s)

We start by cloning FreeBSD Ports from GitHub and pulling a customized Ports collection (maintained internally) into our local FreeBSD Ports repository. We’ll finish by describing the process for pulling updates into the custom tree from each of the two repos.

Creating The Repo

  1. git clone https://www.github.com/freebsd/freebsd-ports.git
  2. cd freebsd-ports
  3. git remote add repo /path/to/local/repo
  4. git pull repo master
  5. cd $new_physical_category/$custom_port && make describe (optional, reports errors if they exist)
  6. run portlint on the port (Verifies correct state of Port)
  7. env PORTSDIR=/path/to/ports sh -e /path/to/ports/Tools/scripts/chkorigin.sh
  8. cd /path/to/combined/ports_tree && make checksubdirs && make index

Updating the Repo

  1. cd /path/to/local/freebsd-ports/clone
  2. git pull origin
  3. git pull repo master

If new physical categories and/or ports are added to the internal repo, steps 5 – 8 above need to be executed. Other changes may or may not require this.

Summary

This procedure creates a combined Ports collection from two sources, FreeBSD’s official Ports collection and a private collection. Once completed, normal operations can be performed on the resulting Ports tree as necessary such as installing applications or creating packages, etc.

One use of this repo is to commit to git and tag each commit as a new version of the custom Ports collection thus making them easily recallable.

Disclaimer

Data and information described on this blog are for informational purposes only. The author of this blog provides no warranty/guarantee, expressed or implied, that this data and information will function as described here. Readers are expected to exercise due diligence when researching, developing, and deploying techniques and methods for use within their environments.

Comments posted are the explicit opinions of the comment poster themselves and does not necessarily reflect the views and opinions of the author of this blog.

Categories: FreeBSD

vBSDcon Wrap-Ups

November 20, 2013 1 comment

vBSDcon Wrap-Ups!

Verisign’s first-ever vBSDcon was a very successful event which was held October 25 – 27, 2013 at the Dulles Hyatt in Herndon, VA.  Speakers and attendees from all over the US and Europe convened at vBSDcon for exclusive networking opportunities, plenary talks, birds of a feathers, and speed geeking.

In the weeks following the conference, several organizations posted follow-ups of the event.  We’d like to encourage you to take a few minutes to read through them at the links below.

  • iXsystems, a vBSDcon Gold sponsor, posted this write-up
  • RootBSD, the vBSDcon Tote sponsor, posted this write-up
  • BSDNow, a weekly BSD news podcast, has a podcast review of vBSDcon here
  • Undeadly.org published a small article based on Henning Brauer and Reyk Floeter’s experiences here.
  • Check out the million+ impression tweets here
  • John-Mark Gurney provided a trip report here
  • Verisign’s own wrap-up, by CTO Burt Kaliski, is here

OpenBSD at vBSDcon October 25 – 27, 2013 in Virginia

October 8, 2013 Leave a comment

OpenBSD at vBSDcon October 25 – 27, 2013 in Virginia

For only USD$75 you can register for vBSDcon hosted by Verisign on October 25 – 27, 2013 in Herndon, VA. That is less than 3 weeks away! If you have not registered yet, it is definitely recommended as vBSDcon will feature a series of roundtable discussions, educational sessions, best practice conversations, and exclusive networking opportunities. Registrations for vBSDcon will be open until October 23, 2013 at http://www.vbsdcon.com/.

vBSDcon will feature developers, Henning Brauer and Reyk Floeter, from the OpenBSD project. Henning and Reyk will be tag teaming a presentation on packet inspection with pf in OpenBSD. pf was introduced in 2001 as an alternative to ipf. Over the years, pf has become a mature, secure, and powerful, yet easy to use high performance packet filter. In this presentation, Henning Brauer will describe some of the features of pf and Reyk Floeter will describe utilizing relayd in conjunction with pf. You can expect to hear about using hooks for transparent proxies, deep packet inspection, socket splicing, NATs, load balancing and more.

Read more about our speakers and their topics, the conference agenda, other activities, and registrations at http://www.vbsdcon.com/. This is an event you will not want to miss. Register now before it’s too late!

Follow @VERISIGN and @hostileaddmin on Twitter for more news and updates on #vBSDcon

Enabling mrepo Support for FreeBSD Packages

October 2, 2013 Leave a comment

Enabling mrepo Support for FreeBSD Packages

mrepo is a content management application for Yum/APT RPM Repositories which does not support FreeBSD packages. While this is not necessarily an elegant solution, it’s enough to enable support for FreeBSD packages and can get one started in deploying a more proper implementation.

Enabling preliminary support for FreeBSD packages involves the following procedure. This procedure assumes repos are available via HTTP at the location where content is downloaded from. We also assume a web server is installed and configured to serve mrepo content. Any discussion involving the configuration of these components is beyond the scope of this post.

  1. Apply the patch below to the mrepo.conf and add some FreeBSD repos
  2. Update the local mrepo repositories
  3. Execute `find` symlinking to the updated content
  4. Setup cronjobs to update repos periodically

Applying The Patch

This patch was generated with GNU patch on RHEL6 and should be applied using GNU patch in the following manner to minimize potential issues. It patches the mrepo script, mrepo.conf, and adds three mrepo repositories. The patch:

diff -u usr/bin/mrepo usr/bin/mrepo.new
--- usr/bin/mrepo	2011-07-01 21:16:09.000000000 -0400
+++ usr/bin/mrepo.new	2013-09-26 12:10:22.072081702 -0400
@@ -214,6 +214,7 @@
         self.lftpcleanup = self.getoption('main', 'lftp-cleanup', 'yes') not in disable
         self.lftpexcldebug = self.getoption('main', 'lftp-exclude-debug', 'yes') not in disable
         self.lftpexclsrpm = self.getoption('main', 'lftp-exclude-srpm', 'yes') not in disable
+        self.lftpincltbz = self.getoption('main', 'lftp-include-tbz', 'yes') not in disable
         self.lftpoptions = self.getoption('main', 'lftp-options', '')
         self.lftpcommands = self.getoption('main', 'lftp-commands', '')
         self.lftpmirroroptions = self.getoption('main', 'lftp-mirror-options', '-c -P')
@@ -1282,6 +1283,8 @@
         mirroropts = mirroropts + ' -X \"*.src.rpm\" -X \"/SRPMS/\"'
     if cf.lftpexcldebug:
         mirroropts = mirroropts + ' -X \"*-debuginfo-*.rpm\" -X \"/debug/\"'
+    if cf.lftpincltbz:
+        mirroropts = mirroropts + ' -I \"*.tbz*\"'
 
     ret = run('%s %s -c \'%s mirror %s %s %s\'' % (cf.cmd['lftp'], opts, cmds, mirroropts, url, path),dryrun=True)
     if ret:
diff -ruN etc/mrepo.conf newetc/mrepo.conf
--- etc/mrepo.conf	2013-09-30 10:57:32.036840555 -0400
+++ newetc/mrepo.conf	2013-09-30 09:08:34.960901787 -0400
@@ -1,4 +1,4 @@
-
+### Configuration file for mrepo
 
 ### The [main] section allows to override mrepo's default settings
 ### The mrepo-example.conf gives an overview of all the possible settings
@@ -23,3 +23,5 @@
 hardlink = no
 shareiso = no
 rhnget-download-all = yes
+
+lftp-include-tbz = yes
diff -ruN etc/mrepo.conf.d/FreeBSD-8-x86_64-dev.conf newetc/mrepo.conf.d/FreeBSD-8-x86_64-dev.conf
--- etc/mrepo.conf.d/FreeBSD-8-x86_64-dev.conf	1969-12-31 19:00:00.000000000 -0500
+++ newetc/mrepo.conf.d/FreeBSD-8-x86_64-dev.conf	2013-09-30 10:59:07.834704096 -0400
@@ -0,0 +1,5 @@
+[FreeBSD-8-x86_64-dev]
+name = FreeBSD 8.x Packages - Dev Repo
+release = 8
+metadata = yum repomd
+dev-repo = $url_to_the_development_repo
diff -ruN etc/mrepo.conf.d/FreeBSD-8-x86_64-prod.conf newetc/mrepo.conf.d/FreeBSD-8-x86_64-prod.conf
--- etc/mrepo.conf.d/FreeBSD-8-x86_64-prod.conf	1969-12-31 19:00:00.000000000 -0500
+++ newetc/mrepo.conf.d/FreeBSD-8-x86_64-prod.conf	2013-09-30 10:59:18.794803601 -0400
@@ -0,0 +1,5 @@
+[FreeBSD-8-x86_64-prod]
+name = FreeBSD 8.x Packages - Production Repo
+release = 8
+metadata = yum repomd
+prod-repo = $url_to_the_production_repo
diff -ruN etc/mrepo.conf.d/FreeBSD-8-x86_64-qa.conf newetc/mrepo.conf.d/FreeBSD-8-x86_64-qa.conf
--- etc/mrepo.conf.d/FreeBSD-8-x86_64-qa.conf	1969-12-31 19:00:00.000000000 -0500
+++ newetc/mrepo.conf.d/FreeBSD-8-x86_64-qa.conf	2013-09-30 10:59:28.171888437 -0400
@@ -0,0 +1,5 @@
+[FreeBSD-8-x86_64-qa]
+name = FreeBSD 8.x Packages - QA Repo
+release = 8
+metadata = yum repomd
+qa-repo = $url_to_the_qa_repo

are applied by executing the following respectively for each patch after the patches have been downloaded:

# cd / && patch < mrepo.patch

Note this patch includes variables that should be replaced by URLs where the repos exist. This URL could be the URL to FreeBSD’s binary package repos, but the size of the repo could cause some issues with this process, especially the step below where the symlinks are generated. My use case involves HTTP repos of home grown FreeBSD packages.

This patch can also be modified to enable support for other file types. For example, tgz and txz files which are supported in FreeBSD’s new pkg framework. which will be default in FreeBSD 10.0-RELEASE. Doing so would enable support for FreeBSD 10 ‘pkg’ files to mrepo as well.

Updating The Repositories

mrepo repositories are updated by executing the patched mrepo script as described here:

# mrepo -ug dev-repo
# mrepo -ug qa-repo
# mrepo -ug prod-repo

These commands update each of the above repos [ included in the patch ] according to srcdir and wwwdir specified in mrepo.conf. Therefore, content downloads to srcdir and wwwdir will be updated in the next step.

Symlink Content

mrepo downloads content from the URLs specified in the mrepo config files located in /etc/mrepo.conf.d/. The FreeBSD-8-x86_64-*.conf repositories in this case. The files are symlinked to from the repo identified by the combination of the value in wwwdir specified in /etc/mrepo.conf and the repo name. This is done by executing the following:

# cd $wwwdir/$repodir && find $srcdir/$repo –type f –name '*tbz' -exec ln –s {} . \;

Where $wwwdir and $srcdir are taken from /etc/mrepo.conf and $repodir is actual directory name of the repo on disk. This step should be executed for each of the repos. Completion of this step means the repos are now available via HTTP.

Configure Cronjobs

It is generally considered a good idea to enable periodic updates of mrepo repos. This can be done in any number of ways, including cronjobs. I choose cronjobs here for simplicity’s sake. Your cronjobs may look similar to the following:

0 0 1 * * mrepo -ug $repo && cd $wwwdir/$repo && find $srcdir/$repo –type f –name ‘*tbz’ -exec ln –s {} . \;

Where $repo is the repo tag specified in it’s config file in /etc/mrepo.conf.d/, $srcdir and $wwwdir are as specified in /etc/mrepo.conf.

Make Content Available Via Cobbler

Some technologists may employ provisioning platforms to distribute OS and package repo contents throughout an environment. I utilize Cobbler in my environments. Therefore, an additional one-time step to create the Cobbler repo is necessary for each repo.

Once the symlinks are created in $wwwdir/$repo, one can add the repo to Cobbler making it available there. This is done by executing:

# cobbler repo add --name="$repo_name" --breed="rsync" --mirror="$wwwdir/$repo"

The new repos will be updated any time a `cobbler reposync` is executed.

Disclaimer

This blog is posted for informational purposes only. Extensive testing is recommended prior to implementing changes discussed here.

vBSDcon Registrations Only Open For 30 More Days!

September 23, 2013 2 comments

vBSDcon Registrations Only Open For 30 More Days!

There are only 30 more days left to register for Verisign’s vBSDcon.  Online registrations will become unavailable October 23, 2013.  For those planning to attend, we encourage you to register soon at http://www.vbsdcon.com/.  You will not want to miss this event.  There will presentations by several well seasoned technologists such as Baptiste Darroussin on the subject of “PkgNG“, a new packaging system for FreeBSD based system such as FreeBSD, PC-BSD, and Dragonfly BSD.

Baptiste has a background in UNIX Systems Engineering and is involved in multiple facets of the FreeBSD project including being a Ports committer for 3 years and a src committer for 2 years. His involvement also includes being a member of the Port management team. PkgNG, a new package management framework for FreeBSD, is one of Baptiste’s primary roles where he is a lead developer.

In addition to plenary speakers, vBSDcon will also feature after conference hours Hacker Lounges and Doc Sprints.  These sessions will be available for the entire BSD communities to include NetBSD, OpenBSD, FreeBSD, and other BSD based distributions to have a collaborative space to work and communicate with one another.  Complimentary wireless internet access will also be available.

We look forward to seeing you all there for this opportunity to come together as a community.  Remember, online registrations will close on October 23, 2013.  Register for vBSDcon at http://www.vbsdcon.com/

vBSDcon Registrations Remain Open, Join Us In Celebrating 20 Years of FreeBSD!

September 19, 2013 Leave a comment

vBSDcon Registrations Still Open!

As many of you are aware, the social aspect of BSD-related conferences is very important and offers opportunities to meet and socialize with one another. Maintaining that tradition, Verisign’s vBSDcon will feature a mid-conference social, brought to you exclusively by Juniper, and will be celebrating 20 years of FreeBSD. We encourage all attendees to join Verisign and Juniper to celebrate this milestone for the FreeBSD project.

Conference activities start on October 25, 2013 at 6:00PM Eastern with a reception dinner hosted by Verisign at the Dulles Hyatt. General conference activities start the following morning with a presentation by David Chisnall, FreeBSD Core Team member, on the migration from GCC to LLVM/Clang within FreeBSD. David Chisnall is a Research Associate at the University of Cambridge, where he works on the interface between languages, operating systems, and hardware. He is also a member of the FreeBSD Core Team and an LLVM/Clang committer. He is the author of several books, including the Definitive Guide to the Xen Hypervisor. He created the current GNUstep implementation of Objective-C and has maintained it for some years, and is now mostly responsible for the C++ stack in FreeBSD, having implemented the ABI library and ported the STL implementation.

We are in high gear planning for vBSDcon 2013 hosted by Verisign at the Dulles Hyatt in Herndon, VA and we are drawing closer by the week with 5 weeks left to register. Registrations are being accepted on the conference web site at http://www.vbsdcon.com/ through October 23, 2013 after which registrations will be taken in person at the event.

Reminder: vBSDcon Registrations Are Open!

September 9, 2013 1 comment

vBSDcon Registrations Still Open!

With approximately six (6) weeks until the event, Registrations for vBSDcon 2013 remain open until October 23rd, 2013.  Register now for this BSD conference scheduled to take place at the Dulles Hyatt in Herndon, VA from October 25 – 27, 2013.

Users and developers from across the BSD communities are encouraged to attend the event intended bring together members of the BSD community for a series of roundtable discussions, educational sessions, best practice conversations, and exclusive networking opportunities.  Those interested in such an opportunity to learn, experience, and meet others involved in the BSD communities should plan to attend vBSDcon 2013.

vBSDcon is proud to bring such prolific speakers like:

vBSDcon is being hosted at the Dulles Hyatt in Herndon, VA making it extremely convenient for attendees who book their room at the venue.  The venue is also just minutes from Dulles International Airport with regular shuttles to/from the hotel and airport terminal during the day.  Breakfast, lunch, and snacks will be provided on-site by the hotel’s on premise restaurant.

The schedule includes a reception dinner at the Dulles Hyatt on the evening of October 25th provided by Verisign and a mid-conference social the following evening celebrating 20 years of FreeBSD.  Space for off hours hacker lounges and doc sprints will be available in the conference facilities with complimentary wireless internet access.  The BSD Certification Group will also be hosting a BSDA certification exam on Saturday evening following the completion of conference activities for the day.

All are invited to take part in this event and are encouraged to register at the vBSDcon web site at http://www.vbsdcon.com/.  Simply click the “Register now” button to begin your registration!  We look forward to meeting you all there!

vBSDcon 2013 Registrations Now Open!

August 12, 2013 Leave a comment

vBSDcon Registrations Now Open!

In April 2013, Verisign announced vBSDcon 2013 to be held October 25 – 27, 2013 in Dulles, VA. The conference, formatted to resemble an unConference concept, will feature speakers such as David Chisnall, Luigi Rizzo, Baptiste Daroussin, Henning Brauer, Reyk Floeter, and others.  vBSDcon will include events like hacker lounges, doc sprints, BSDA exams, and a mid-conference social*.

In these most recent months, they have been developing the vBSDcon conference website hosted at http://www.vbsdcon.com/.  It includes full details surrounding the schedule, agenda, and speakers for vBSDcon.  The most recent addition to the conference website is that registrations are now open!

* Schedule is subject to change without notice, The BSDA exams are hosted by the BSD Certification group and not an official part of vBSDcon.

vBSDcon Website Update…

July 22, 2013 Leave a comment

vBSDcon Website Update…

In April 2013, Verisign announced vBSDcon, a BSD-related conference, in Dulles, VA to occur October 25 – 27, 2013. In the weeks following the announcement, the vBSDcon website was activated with preliminary details on the dates and location of the event.  This past weekend, the next phase of development of the website was published.  The updated website contains a detailed conference agenda, speaker biographies, and descriptions of speaker’s topics, and more.  The website website can be viewed at http://www.vbsdcon.com/.

The next phase of development will include a map detailing conference facilities/location, social event location(s), and local points of interest, restaurants, and recreational activities.  Also slated for the next phase is the addition of a sponsor page and attendee registrations.  So, check the site often and look for registrations to open up in the coming weeks!

BSDCan 2013

May 27, 2013 1 comment
I blogged my experience at MeetBSD 2012 a few months ago and recently attended BSDCan for the first time.  Since I blogged my experience at MeetBSD, I also wanted to blog my experience at BSDCan.

I attended BSDCan as an attendee and conference organizer.  I recently embarked on journey I never thought I would.  I am a co-chair for a conference.  vBSDcon is the first conference or large event that I’ve organized (apart from my wedding when I married the woman who continually encourages me to grow).  Therefore, my perspective of the conference covered multiple facets.

Conference Format

The conference covered two days and was preceded by 2 days of developer summit sessions, a vendor summit, and tutorials. The main conference consisted of 3 tracks (hacking, embedded, and system administration) meaning that there were 3 presenters at any one time during speaking sessions.

I’ll add that all conferences have an additional track which may or may not be planned…the social track. The social track consists of time spent in the halls between presentations discussing projects with others. It includes time spent at restaurants after the day’s conference activities. It includes time spent at the “big social event” many BSD-related conferences have.

One drawback of multiple tracks is that often there are multiple presentations occurring simultaneously one may wish to attend. This certainly was the case with me a time or two.

Tutorials

Tutorials are educational presentations designed to educate attendees on some of the nuts and bolts of a project and/or technology. They are typically 3 hours in length with an instructor/presenter who is considered an expert on the topic.

BSDCan 2013 had four tutorials scheduled. Of the four, I attended two:

DNSSEC: Theory, Troubleshooting, and Deployment With BIND

Presented by Michael W. Lucas, this tutorial encompassed operating DNSSEC in BIND. He covered the operational concepts of DNSSEC and followed up by describing BIND configurations to support DNSSEC. He introduced the various binaries used in troubleshooting various aspects of DNSSEC and BIND configurations and operations.

Making FreeBSD Ports

Presented by Gábor Páli, this tutorial covered creating a FreeBSD Port from scratch. Gábor included many details and explanations of various aspects of creating a Port including many common variables and macros to use in Makefiles. He also expanded on the content of other potential files that can be created and how to generate those files.

Vendor Summit

The vendor summit is an opportunity for FreeBSD committers and vendors to collaborate together on setting priorities for future projects. George Neville-Neil headed up the session focusing on userland space in FreeBSD. Many areas where vendors expressed needs included development and user related desktop projects, most of which seemed to circle back to the first item on the list…Java.

It was made clear that vendors are interested in having a cohesive and functional Java implementation available. By doing so, it will enable forward progress on the implementation of many other userland applications.

Conference Day 1

Presentations

Conference Day 2

Presentations

These are the presentations I attended on Day 2:

Impressions

Dan Langille, BSDCan organizer, did an excellent job of ensuring the conference went according to plan from the start of the Developer’s Summit through the tutorials and into the conference activities. The logistics of ensuring wi-fi access was available, providing power strips and power cables enabling users to take notes on laptops, and providing food, among the many other tasks he must attend to. It really is an impressive feat.

All speakers created well thought out presentations all the way from the tutorials through the closing session. The content was relevant and applicable to the track and current trends in the industry. Speakers sufficiently filled their allotted time slot w/ ample time for Q&A. I found several presentations very interesting and relevant to the work I perform.

Overall Experience

The overall experience at BSDCan 2013 was extremely positive. I was impressed with the execution of the various activities, including those in the unofficial “social track”. This conference provided excellent opportunities to absorb information, collaborate with others, and socialize and meet new people.

This conference was an excellent opportunity for me to gleam concepts and ideas for potential inclusion at vBSDcon hosted by Verisign in October 2013.