Verisign Announces vBSDcon 2015

May 1, 2015 1 comment

Verisign has recently announced the dates for a second vBSDcon to occur September 11 – 13, 2015 in Reston, VA. vBSDcon is technical, BSD-based conference for users, developers, engineers, and innovators working with BSD-based operating systems such as FreeBSD, OpenBSD, NetBSD, and more…

The program is being expanded in 2015 to colocate a FreeBSD Developer’s Summit the day preceding main conference activities. Additionally, space will be allotted for use of Hacker Lounges and Doc Sprints. These self-moderated sessions are open time during the off hours of the conference for use by members of all the BSD communities to collaborate and work on varying projects.

The inaugural vBSDcon was a resounding success and included speakers from the FreeBSD and OpenBSD communities.  It also featured prestigious sponsors including The FreeBSD Foundation, who is returning as a Developer Summit sponsor in 2015, RootBSD, Daemon Security, iXsystems, HP, Juniper, and CDW.

Starting A Daemon via daemon(8) in FreeBSD

January 27, 2015 1 comment

Starting A Daemon via daemon(8) in FreeBSD

Daemonizing an application that must persist across reboots while including native process management capability is easily accomplished by combining the functions of rc(8) scripts and commands with daemon(8), a utility that detaches itself from the controlling terminal and executing a program. For example, a service written in Python implementing an HTTP API front-end.

Combing these two utilities enables automated process initialization during system boot and provides the ability to stop, start, and restart services manually using either the rc(8) script or service(8). For example:

/usr/local/etc/rc.d/httpapi start

or

service httpapi start

Create an rc(8) script in /usr/local/etc/rc.d with a name descriptive of the application. httpapi is a descriptive name in this case. Writing the rc(8) script is covered extensively well in the “Practical rc.d scripting in BSD” article on FreeBSD.org and is a very good read. Those reading this blog are encouraged to go read that article though, for expediency, a basic implementation may look very similar to:


#!/bin/sh

# REQUIRE: LOGIN

. /etc/rc.subr

name=httpapi
rcvar=httpapi_enable
command=/usr/local/bin/httpapi
command_interpreter=python
httpapi_user=httpapi
start_cmd=”/usr/sbin/daemon -u $httpapi_user $command”

load_rc_config $name
run_rc_command “$1”

The value of the key rcvar is used to automate the startup of the daemon at system boot within /etc/rc.conf. Append a line to the file containing a combination of this value as key a key with a boolean “YES” or “NO” similarly to:

echo "httpapi_enable=\"YES\"" >> /etc/rc.conf

Disclaimer

Data and information described on this blog are for informational purposes only. The author of this blog provides no warranty/guarantee, expressed or implied, that this data and information will function as described here. Readers are expected to exercise due diligence when researching, developing, and deploying techniques and methods for use within their environments.

Comments posted are the explicit opinions of the comment poster themselves and does not necessarily reflect the views and opinions of the author of this blog.

Categories: FreeBSD

QEMU: Convert non-Sparse Image to qcow/qcow2 Sparse Image

December 15, 2014 Leave a comment

Convert non-Sparse Image to qcow/qcow2 Sparse Image

While this is not always ideal, I’ve found it handy to build qcow/qcow2 images (for use in OpenStack) by converting a vmdk on FreeBSD. Often times, VMWare images are created with thick provisioning meaning that the disk consumed on the hypervisor will equal the disk allocated to the virtual though there may be an overwhelming quantity of contiguous zero bytes.  For example, creating a thick provisioned disk for a VM of, say, 20GB will result in approximately 20GB being consumed on the hypervisor’s disk even though, perhaps, only 50% of that is in use. The remaining 50% will be zero’d bytes and, while technically empty, still consuming space.

VMDK’s can be converted to qcow/qcow2 using qemu-img(1) similarly to:

qemu-img convert -f vmdk -O qcow2 freebsd-10.0.0-amd64.vmdk freebsd-10.0.0-amd64.qcow2

Excellent, we now have qcow2 image suitable for use within OpenStack. However, the size of this file is likely to be the same or very similar to the original vmdk. This can be taken a step further to significantly reduce the amount of space the file consumes using a sparse file.

Using the ‘-S’ command line option enables us to ensure the output file becomes sparse, therefore freeing the space it consumed with a series of zero bytes. This option, while it defaults to 4k, accepts an argument* (size) permitting users to specify the quantity of bytes which must be zeros before it can be that portion is made “sparse”.

The resulting command line is very similar to the above, except that it includes ‘-S’ and the argument, for example:

qemu-img convert -f vmdk -O qcow2 -S 4k freebsd-10.0.0-amd64.vmdk freebsd-10.0.0-amd64.qcow2

* NOTE: if installed from FreeBSD Ports, the argument was required despite having a default defined.

Disclaimer

Data and information described on this blog are for informational purposes only. The author of this blog provides no warranty/guarantee, expressed or implied, that this data and information will function as described here. Readers are expected to exercise due diligence when researching, developing, and deploying techniques and methods for use within their environments.

Comments posted are the explicit opinions of the comment poster themselves and does not necessarily reflect the views and opinions of the author of this blog.

Categories: Technical Miscellany

Troubleshooting Large, Stalling git/ssh Transfers

July 14, 2014 1 comment

Why is git/ssh stalling?

While working with the freebsd-ports repo on github in order to track and combine it with an internal remote repo, it was found to apparently stall during a git push to the internal remote repo. Thinking this was likely a fluke in the network or something, the process was re-executed, but exhibited the same behavior. The OS in this case is FreeBSD 10.0-RELEASE-p6.

Troubleshooting

Identifying the problem started by observing the git process in top which was in a “piperd” wait state. An email thread on freebsd-questions@[1] suggested the process was waiting on a read from a pipe. I only needed to identify the process it was waiting for.

Using lsof, according to the same reference, one could identify the offending process by matching the hex value of the pipe file descriptor to the piped process, in this case ssh. Attaching truss to ssh showed that it had apparently stalled on a getsockopt() operation.

A web search turned up related results[2] (found in References below) which indicated the likely problem was a bug in OpenSSH related to TcpRcvBufPoll. Explicitly disabling this in sshd_config and restarting the service did appear to work around the problem. Upon re-execution of my git push, the process didn’t stall and completed successfully.

Oddly enough, after working around this, I was unable to duplicate the behavior on the same host later. Therefore, there is some doubt that this was indeed the problem, but I post this anyway as it did appear to successfully work around it.

References

[1] FreeBSD Mailing List Archives
[2] stackoverflow post

Disclaimer

Data and information described on this blog are for informational purposes only. The author of this blog provides no warranty/guarantee, expressed or implied, that this data and information will function as described here. Readers are expected to exercise due diligence when researching, developing, and deploying techniques and methods for use within their environments.

Comments posted are the explicit opinions of the comment poster themselves and does not necessarily reflect the views and opinions of the author of this blog.

Manual Package Builds For FreeBSD < 10.0-RELEASE

May 28, 2014 1 comment

Just wanted to link to an older blog post I found useful recently…

Given the current state of migration from legacy pkg_* tools to pkg(8) within FreeBSD, the content here is old, but still useful for those that haven’t migrated to 10.x (or newer) or the new pkg(8) framework.

Check out Troy’s Unix Space post on package builds on blogspot.

Disclaimer

Data and information described on this blog are for informational purposes only. The author of this blog provides no warranty/guarantee, expressed or implied, that this data and information will function as described here. Readers are expected to exercise due diligence when researching, developing, and deploying techniques and methods for use within their environments.

Comments posted are the explicit opinions of the comment poster themselves and does not necessarily reflect the views and opinions of the author of this blog.

Categories: FreeBSD

BSDCan 2014 Travel Woes

May 16, 2014 3 comments
I wasn’t going to post anything about my travel woes to Ottawa, ON for BSDCan 2014, but upon receiving the revised itinerary after the 3rd flight cancellation, I decided it was too humorous to pass up…

Originally scheduled to fly out on Wednesday via United, I arrived at the airport, parked my vehicle, removed my bags, and proceeded to walk to the main terminal.  My phone vibrated while walking to United’s check-in counters…I just received a flight cancellation notice citing “severe weather conditions in our route network” despite there being little weather in the areas of Dulles and Ottawa that day.  A United representative rescheduled the flight the following morning which was subsequently cancelled for the same reason…still no real weather to be seen, but there was certainly weather on the way for Thursday evening into Friday.

I contacted the travel agent following the 2nd cancellation and discussed two options;  A connecting flight with a 4 hour layover that day which ended up being cancelled or a non-stop flight Friday morning.  Seeing as how the total time on the former flight would have been 8 – 10 hours, I opted for the latter, shorter, 1.5 hour flight Friday morning, just in time to attend most of BSDCan’s main conference activities.

I awoke Friday morning to a deluge of rain thinking the flight would certainly be cancelled, but the flight status, according to United, was “on schedule”.  I got up and continued my daily morning routine intending to make the drive to the airport while checking the phone periodically for flight status updates.  The phone vibrated as I was preparing to leave home with an email update that the flight was cancelled.  It was the cancellation I had been expecting so, instead of driving to the airport, I continued into the office.

Shortly after arriving at the office, I received an email from United with the revised itinerary:

The irony in the revised itinerary is two-fold in that I depart Dulles to Ottawa on the same plane I was originally scheduled to return to Dulles from Ottawa on and that United thinks it’s ok for me to fly from Dulles to Ottawa, stay on the plane, and fly back to Dulles approximately 30 minutes later.

Categories: Conferences & Summits

FreeBSD 10.0: release.sh mapped

February 24, 2014 1 comment

FreeBSD‘s release.sh is a shell script introduced in FreeBSD 9.x whose purpose is to automate FreeBSD release building from source.  This post maps the release.sh into a table of variables and a flowchart describing the program flow and is based on release.sh as it exists in releng/10.0 on FreeBSD‘s subversion or GitHub.

Variables

The following table describes each available variable. The “Default” column contains strings which each variable is set to, with the exception of “Unset” and “NULL”. Unset variables are uninitialized while NULL variables are initialized with no value. Required variables are denoted with an asterisk (*) immediately appended to the name.

Variable Purpose Default
PATH* The environment’s $PATH /bin:/sbin:/usr/bin:
/usr/sbin:/usr/local/bin
CHROOTDIR* The relative or fully qualified path of the chroot where non-contaminated distributions are built /scratch
SVNROOT* The protocol and address of the subversion repository from which sources, docs, and ports are checked out from svn://svn.freebsd.org
SRCBRANCH* The source branch to checkout, compile, and install base/head@rHEAD
DOCBRANCH The doc branch to checkout, compile, and install docs/head@rHEAD
PORTBRANCH The ports branch to checkout, compile, and install ports/head@rHEAD
SRC_FORCE_CHECKOUT Enables use of “–force” with the svn command line NULL
MAKE_CONF* The path to the make.conf file to implement in the chroot for release builds /dev/null
SRC_CONF* The path to the src.conf file to implement in the chroot for release builds /dev/null
NCPU* Derives the number of CPU cores available
WORLD_FLAGS make buildworld make flags. Sets -j to $NCPU Unset
KERNEL_FLAGS make buildkernel make flags. Sets -j to $NCPU / 2 Unset
MAKE_FLAGS* Set’s the -s flag preventing the commands from being echoed to STDOUT -s
KERNEL* Specify the kernel to build with the distribution GENERIC
WITH_DVD Enables the generation of a DVD ISO of the resulting image NULL
RELEASECONF Enables operators/engineers to specify a custom release.conf on the command line Unset
NOPORTS If set, prevents the inclusion of a ports tree in the resulting distribution NULL
NODOC If set, prevents the inclusion of a docs tree in the resulting distribution NULL
DOCPORTS If NOPORTS and/or NODOC are unset, they must not pass to make as variables. The release Makefile verifies definedness of NOPORTS/NODOC variables instead of their values. NULL
CONF_FILES* The aggregated build-time flags based upon variables defined within this file, unless overridden by release.conf. In most cases, these will not need to be changed. __MAKE_CONF=${MAKE_CONF} SRCCONF=${SRC_CONF}
__MAKE_CONF Permits the specification of an alternate make.conf Unset
SRCCONF Equal to SRC_CONF /dev/null
TARGET A make variable for cross-compiling releases
TARGET_ARCH A make variable for cross-compiling releases
ARCH_FLAGS Variable specifying architecture dependent built options
CHROOT_WMAKEFLAGS* make buildworld flags for the chroot environment ${MAKE_FLAGS} ${WORLD_FLAGS} ${CONF_FILES}
CHROOT_IMAKEFLAGS* make installworld flags for the chroot environment ${CONF_FILES}
CHROOT_DMAKEFLAGS* make distribution flags for the chroot environment ${CONF_FILES}
RELEASE_WMAKEFLAGS* make buildworld flags for the release inside the chroott ${MAKE_FLAGS} ${WORLD_FLAGS} ${ARCH_FLAGS} ${CONF_FILES}
RELEASE_KMAKEFLAGS* make buildkernel flags for the release inside the chroot ${MAKE_FLAGS} ${KERNEL_FLAGS} KERNCONF=${KERNEL} ${ARCH_FILES} ${CONF_FILES}
RELEASE_RMAKEFLAGS* make release build flags inside the chroot ${ARCH_FILES} KERNCONF=${KERNEL} ${CONF_FILES} ${DOCPORTS}
WITH_DVD=${WITH_DVD}
FORCE_SRC_KEY Enables use of “–force” with the svn command line NULL
RELSTRING* The release string as returned from uname -s Unset
PBUILD_FLAGS Specifies port build flags for docproj Unset
_OSVERSION
OSVERSION
Identifies the OS release date and is included in PBUILD_FLAGS for the docproj port Unset

Flowchart

The following chart describes the release.sh program flow:

Disclaimer

Data and information described on this blog are for informational purposes only. The author of this blog provides no warranty/guarantee, expressed or implied, that this data and information will function as described here. Readers are expected to exercise due diligence when researching, developing, and deploying techniques and methods for use within their environments.

Comments posted are the explicit opinions of the comment poster themselves and does not necessarily reflect the views and opinions of the author of this blog.

Categories: FreeBSD
Follow

Get every new post delivered to your Inbox.

Join 307 other followers