Archive

Archive for the ‘Technical Miscellany’ Category

Internet Privacy: Who Do You Trust?

April 8, 2017 Leave a comment

Who do you trust?

The rollback of the not yet effective USA FCC regulations enacted during the Obama administration poses an interesting dilemma. With the regulations set to take effect in 2017, user’s were/are still subject to whatever unregulated behavior an ISP chooses to exercise. In essence, all internet users are vulnerable to the whims of their ISP. Therefore, the rollback of these regulations does not materially impact Internet user behavior or activity at present.

These FCC regulations would have enabled users to trust that their ISPs, at least to a certain degree, are complying with privacy regulations and keeping much of the data generated by user’s activity private. While this may be an onerous assumption to make, the regulations allay fears of nefarious behavior by ISPs with user’s data.

Coupled with Vault 7 revelations and the FCC regulation rollback, many are concerned about their internet privacy and are clamoring for an answer. Predictably, paid VPN services are being discussed as a means to prevent snooping by user’s ISPs. This is an effective countermeasure achieving that specific goal, but is only a single tool out of many that should be deployed in parallel to minimize one’s exposure.

Deploying VPN services carries with it it’s own set of complications and considerations. Chief among these is trust as one is essentially changing the trust relationship with their ISP in favor of an another entity further along the line, but are prospective VPN services worthy of this? VPN providers have unfettered access to any traffic traversing their network. Additionally, they — and incidentally you — may be subject to laws and regulations of the prevailing authorities/jurisdiction, which may or may not be friendly to the end goal. Even if hosted outside the borders of the US, these services are subject to higher authorities. Can they be trusted? The answer is not clear and concise as it depends largely on the user’s desires.

The next logical leap is to consider self-hosting and operating one’s own VPN endpoint. This removes trust from all providers placing it with the user. The user controls the VPN, its configuration, and operation regardless of the jurisdiction within which it may be deployed. The user can control user registration and records, logging operations and scope, and log retention policies [generally within the constraints of the prevailing jurisdiction].

Even if one chooses to self host a VPN service — particularly within the US — consider the service on which the VPN is deployed may potentially be legally defined as an ISP. After all, they do provide a service on the Internet and can conceivably engage in the same behavior that is feared ISPs such as Verizon or Comcast will engage in. This invalidates the trust relationship with a broader definition of an ISP and/or common carrier. Obviously, the landscape changes with varying jurisdictions, whether US-based or international.

So, who do you trust?

Advertisements
Categories: Technical Miscellany

QEMU: Convert non-Sparse Image to qcow/qcow2 Sparse Image

December 15, 2014 Leave a comment

Convert non-Sparse Image to qcow/qcow2 Sparse Image

While this is not always ideal, I’ve found it handy to build qcow/qcow2 images (for use in OpenStack) by converting a vmdk on FreeBSD. Often times, VMWare images are created with thick provisioning meaning that the disk consumed on the hypervisor will equal the disk allocated to the virtual though there may be an overwhelming quantity of contiguous zero bytes.  For example, creating a thick provisioned disk for a VM of, say, 20GB will result in approximately 20GB being consumed on the hypervisor’s disk even though, perhaps, only 50% of that is in use. The remaining 50% will be zero’d bytes and, while technically empty, still consuming space.

VMDK’s can be converted to qcow/qcow2 using qemu-img(1) similarly to:

qemu-img convert -f vmdk -O qcow2 freebsd-10.0.0-amd64.vmdk freebsd-10.0.0-amd64.qcow2

Excellent, we now have qcow2 image suitable for use within OpenStack. However, the size of this file is likely to be the same or very similar to the original vmdk. This can be taken a step further to significantly reduce the amount of space the file consumes using a sparse file.

Using the ‘-S’ command line option enables us to ensure the output file becomes sparse, therefore freeing the space it consumed with a series of zero bytes. This option, while it defaults to 4k, accepts an argument* (size) permitting users to specify the quantity of bytes which must be zeros before it can be that portion is made “sparse”.

The resulting command line is very similar to the above, except that it includes ‘-S’ and the argument, for example:

qemu-img convert -f vmdk -O qcow2 -S 4k freebsd-10.0.0-amd64.vmdk freebsd-10.0.0-amd64.qcow2

* NOTE: if installed from FreeBSD Ports, the argument was required despite having a default defined.

Disclaimer

Data and information described on this blog are for informational purposes only. The author of this blog provides no warranty/guarantee, expressed or implied, that this data and information will function as described here. Readers are expected to exercise due diligence when researching, developing, and deploying techniques and methods for use within their environments.

Comments posted are the explicit opinions of the comment poster themselves and does not necessarily reflect the views and opinions of the author of this blog.

Categories: Technical Miscellany

Troubleshooting Large, Stalling git/ssh Transfers

July 14, 2014 1 comment

Why is git/ssh stalling?

While working with the freebsd-ports repo on github in order to track and combine it with an internal remote repo, it was found to apparently stall during a git push to the internal remote repo. Thinking this was likely a fluke in the network or something, the process was re-executed, but exhibited the same behavior. The OS in this case is FreeBSD 10.0-RELEASE-p6.

Troubleshooting

Identifying the problem started by observing the git process in top which was in a “piperd” wait state. An email thread on freebsd-questions@[1] suggested the process was waiting on a read from a pipe. I only needed to identify the process it was waiting for.

Using lsof, according to the same reference, one could identify the offending process by matching the hex value of the pipe file descriptor to the piped process, in this case ssh. Attaching truss to ssh showed that it had apparently stalled on a getsockopt() operation.

A web search turned up related results[2] (found in References below) which indicated the likely problem was a bug in OpenSSH related to TcpRcvBufPoll. Explicitly disabling this in sshd_config and restarting the service did appear to work around the problem. Upon re-execution of my git push, the process didn’t stall and completed successfully.

Oddly enough, after working around this, I was unable to duplicate the behavior on the same host later. Therefore, there is some doubt that this was indeed the problem, but I post this anyway as it did appear to successfully work around it.

References

[1] FreeBSD Mailing List Archives
[2] stackoverflow post

Disclaimer

Data and information described on this blog are for informational purposes only. The author of this blog provides no warranty/guarantee, expressed or implied, that this data and information will function as described here. Readers are expected to exercise due diligence when researching, developing, and deploying techniques and methods for use within their environments.

Comments posted are the explicit opinions of the comment poster themselves and does not necessarily reflect the views and opinions of the author of this blog.

Enabling mrepo Support for FreeBSD Packages

October 2, 2013 Leave a comment

Enabling mrepo Support for FreeBSD Packages

mrepo is a content management application for Yum/APT RPM Repositories which does not support FreeBSD packages. While this is not necessarily an elegant solution, it’s enough to enable support for FreeBSD packages and can get one started in deploying a more proper implementation.

Enabling preliminary support for FreeBSD packages involves the following procedure. This procedure assumes repos are available via HTTP at the location where content is downloaded from. We also assume a web server is installed and configured to serve mrepo content. Any discussion involving the configuration of these components is beyond the scope of this post.

  1. Apply the patch below to the mrepo.conf and add some FreeBSD repos
  2. Update the local mrepo repositories
  3. Execute `find` symlinking to the updated content
  4. Setup cronjobs to update repos periodically

Applying The Patch

This patch was generated with GNU patch on RHEL6 and should be applied using GNU patch in the following manner to minimize potential issues. It patches the mrepo script, mrepo.conf, and adds three mrepo repositories. The patch:

diff -u usr/bin/mrepo usr/bin/mrepo.new
--- usr/bin/mrepo	2011-07-01 21:16:09.000000000 -0400
+++ usr/bin/mrepo.new	2013-09-26 12:10:22.072081702 -0400
@@ -214,6 +214,7 @@
         self.lftpcleanup = self.getoption('main', 'lftp-cleanup', 'yes') not in disable
         self.lftpexcldebug = self.getoption('main', 'lftp-exclude-debug', 'yes') not in disable
         self.lftpexclsrpm = self.getoption('main', 'lftp-exclude-srpm', 'yes') not in disable
+        self.lftpincltbz = self.getoption('main', 'lftp-include-tbz', 'yes') not in disable
         self.lftpoptions = self.getoption('main', 'lftp-options', '')
         self.lftpcommands = self.getoption('main', 'lftp-commands', '')
         self.lftpmirroroptions = self.getoption('main', 'lftp-mirror-options', '-c -P')
@@ -1282,6 +1283,8 @@
         mirroropts = mirroropts + ' -X \"*.src.rpm\" -X \"/SRPMS/\"'
     if cf.lftpexcldebug:
         mirroropts = mirroropts + ' -X \"*-debuginfo-*.rpm\" -X \"/debug/\"'
+    if cf.lftpincltbz:
+        mirroropts = mirroropts + ' -I \"*.tbz*\"'
 
     ret = run('%s %s -c \'%s mirror %s %s %s\'' % (cf.cmd['lftp'], opts, cmds, mirroropts, url, path),dryrun=True)
     if ret:
diff -ruN etc/mrepo.conf newetc/mrepo.conf
--- etc/mrepo.conf	2013-09-30 10:57:32.036840555 -0400
+++ newetc/mrepo.conf	2013-09-30 09:08:34.960901787 -0400
@@ -1,4 +1,4 @@
-
+### Configuration file for mrepo
 
 ### The [main] section allows to override mrepo's default settings
 ### The mrepo-example.conf gives an overview of all the possible settings
@@ -23,3 +23,5 @@
 hardlink = no
 shareiso = no
 rhnget-download-all = yes
+
+lftp-include-tbz = yes
diff -ruN etc/mrepo.conf.d/FreeBSD-8-x86_64-dev.conf newetc/mrepo.conf.d/FreeBSD-8-x86_64-dev.conf
--- etc/mrepo.conf.d/FreeBSD-8-x86_64-dev.conf	1969-12-31 19:00:00.000000000 -0500
+++ newetc/mrepo.conf.d/FreeBSD-8-x86_64-dev.conf	2013-09-30 10:59:07.834704096 -0400
@@ -0,0 +1,5 @@
+[FreeBSD-8-x86_64-dev]
+name = FreeBSD 8.x Packages - Dev Repo
+release = 8
+metadata = yum repomd
+dev-repo = $url_to_the_development_repo
diff -ruN etc/mrepo.conf.d/FreeBSD-8-x86_64-prod.conf newetc/mrepo.conf.d/FreeBSD-8-x86_64-prod.conf
--- etc/mrepo.conf.d/FreeBSD-8-x86_64-prod.conf	1969-12-31 19:00:00.000000000 -0500
+++ newetc/mrepo.conf.d/FreeBSD-8-x86_64-prod.conf	2013-09-30 10:59:18.794803601 -0400
@@ -0,0 +1,5 @@
+[FreeBSD-8-x86_64-prod]
+name = FreeBSD 8.x Packages - Production Repo
+release = 8
+metadata = yum repomd
+prod-repo = $url_to_the_production_repo
diff -ruN etc/mrepo.conf.d/FreeBSD-8-x86_64-qa.conf newetc/mrepo.conf.d/FreeBSD-8-x86_64-qa.conf
--- etc/mrepo.conf.d/FreeBSD-8-x86_64-qa.conf	1969-12-31 19:00:00.000000000 -0500
+++ newetc/mrepo.conf.d/FreeBSD-8-x86_64-qa.conf	2013-09-30 10:59:28.171888437 -0400
@@ -0,0 +1,5 @@
+[FreeBSD-8-x86_64-qa]
+name = FreeBSD 8.x Packages - QA Repo
+release = 8
+metadata = yum repomd
+qa-repo = $url_to_the_qa_repo

are applied by executing the following respectively for each patch after the patches have been downloaded:

# cd / && patch < mrepo.patch

Note this patch includes variables that should be replaced by URLs where the repos exist. This URL could be the URL to FreeBSD’s binary package repos, but the size of the repo could cause some issues with this process, especially the step below where the symlinks are generated. My use case involves HTTP repos of home grown FreeBSD packages.

This patch can also be modified to enable support for other file types. For example, tgz and txz files which are supported in FreeBSD’s new pkg framework. which will be default in FreeBSD 10.0-RELEASE. Doing so would enable support for FreeBSD 10 ‘pkg’ files to mrepo as well.

Updating The Repositories

mrepo repositories are updated by executing the patched mrepo script as described here:

# mrepo -ug dev-repo
# mrepo -ug qa-repo
# mrepo -ug prod-repo

These commands update each of the above repos [ included in the patch ] according to srcdir and wwwdir specified in mrepo.conf. Therefore, content downloads to srcdir and wwwdir will be updated in the next step.

Symlink Content

mrepo downloads content from the URLs specified in the mrepo config files located in /etc/mrepo.conf.d/. The FreeBSD-8-x86_64-*.conf repositories in this case. The files are symlinked to from the repo identified by the combination of the value in wwwdir specified in /etc/mrepo.conf and the repo name. This is done by executing the following:

# cd $wwwdir/$repodir && find $srcdir/$repo –type f –name '*tbz' -exec ln –s {} . \;

Where $wwwdir and $srcdir are taken from /etc/mrepo.conf and $repodir is actual directory name of the repo on disk. This step should be executed for each of the repos. Completion of this step means the repos are now available via HTTP.

Configure Cronjobs

It is generally considered a good idea to enable periodic updates of mrepo repos. This can be done in any number of ways, including cronjobs. I choose cronjobs here for simplicity’s sake. Your cronjobs may look similar to the following:

0 0 1 * * mrepo -ug $repo && cd $wwwdir/$repo && find $srcdir/$repo –type f –name ‘*tbz’ -exec ln –s {} . \;

Where $repo is the repo tag specified in it’s config file in /etc/mrepo.conf.d/, $srcdir and $wwwdir are as specified in /etc/mrepo.conf.

Make Content Available Via Cobbler

Some technologists may employ provisioning platforms to distribute OS and package repo contents throughout an environment. I utilize Cobbler in my environments. Therefore, an additional one-time step to create the Cobbler repo is necessary for each repo.

Once the symlinks are created in $wwwdir/$repo, one can add the repo to Cobbler making it available there. This is done by executing:

# cobbler repo add --name="$repo_name" --breed="rsync" --mirror="$wwwdir/$repo"

The new repos will be updated any time a `cobbler reposync` is executed.

Disclaimer

This blog is posted for informational purposes only. Extensive testing is recommended prior to implementing changes discussed here.

Apache/Plesk php Exploit

August 7, 2013 Leave a comment

While reviewing web server logs recently, I spotted the following attack…

216.237.113.27 – – [07/Aug/2013:08:12:14 -0400] “POST
/%70%68%70%70%61%74%68/%70%68%70?%2D%64+%61%6C%6C%6F
%77%5F%75%72%6C%5F%69%6E%63%6C%75%64%65%3D%6F%6E+%2D
%64+%73%61%66%65%5F%6D%6F%64%65%3D%6F%66%66+%2D%64+
%73%75%68%6F%73%69%6E%2E%73%69%6D%75%6C%61%74%69%6F
%6E%3D%6F%6E+%2D%64+%64%69%73%61%62%6C%65%5F%66%75
%6E%63%74%69%6F%6E%73%3D%22%22+%2D%64+%6F%70%65%6E
%5F%62%61%73%65%64%69%72%3D%6E%6F%6E%65+%2D%64+%61
%75%74%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D
%70%68%70%3A%2F%2F%69%6E%70%75%74+%2D%6E HTTP/1.1”
404 209 “-” “-”

Which decodes to:

/phppath/php?-d allow_url_include=on -d safe_mode=off
-d suhosin.simulation=on -d disable_functions=”” -d
open_basedir=none -d auto_prepend_file=php://input -n

Googling turned up a few relevant results suggesting this is an attack on Plesk, a commercial web hosting automation program. Parallels, the company who owns Plesk, published a knowledge base article on the vulnerability describing how to determine if a deployment is vulnerable and how to patch the system.

One aspect of a vulnerable deployment includes an Apache configuration utilizing the following ScriptAlias configuration:

ScriptAlias /phppath/ “/usr/bin/”

For those running Plesk or others with the above configuration, it is recommended that your system be patched accordingly. Check the references below for more detailed information regarding this vulnerability.

References

Disclaimer

This blog is posted for informational purposes only. Extensive testing is recommended prior to implementing changes discussed here.

Categories: Technical Miscellany

Images are back…

July 20, 2013 Leave a comment

A few weeks ago, the node that hosts hostileadmin.com experienced a catastrophic hardware failure.  The hostileadmin blog was affected by this outage in the area of images and downloadable files.

Consequently, I procured virtual hosting services with RootBSD and have moved data from the old node to the VPS service.  I am pleased to announce hostileadmin blog images and downloadable files are back!

I apologize for any inconvenience this may have caused.

Categories: Technical Miscellany

Upcoming Changes to hostileadmin Blog

May 31, 2013 Leave a comment

hostileadmin.com has experienced a hardware failure in a colocation facility that is geographically difficult to reach in a timely manner.  The hostileadmin blog is affected by this outage in the area of images and downloaded files.  Thus, blog posts containing images and/or other downloadable files are incomplete.

As a result of this outage, I have procured virtual hosting services with RootBSD and will begin hosting my images and files from this point forward on the new service.  At an undetermined point in the future, the data on the failed hardware will be recovered and moved to the RootBSD virtual hosting service.

I have purchased a level of service that affords me the ability to move the majority of my disparate services into a single, more reliable environment.  This transition will take place in phases, but will have minimal impact on the blog, which is my most visible area of work at this time.

I apologize for any inconvenience and hope to have services fully restored in the coming weeks.

Categories: Technical Miscellany