Home > FreeBSD > Disk I/O Performance Under Filesystem Journaling on FreeBSD 10.3

Disk I/O Performance Under Filesystem Journaling on FreeBSD 10.3

December 16, 2016 Leave a comment Go to comments

Overview

This post describes disk I/O performance capabilities of the HP DL360p G8 in terms of filesystem journaling mechanisms on FreeBSD 10.3.

References

  1. https://people.freebsd.org/~kris/scaling/Help_my_system_is_slow.pdf
  2. http://docplayer.net/2690085-Hp-smart-array-controllers-and-basic-raid-performance-factors.html
  3. https://dom.as/2009/03/11/iostat/

Hardware

  • Hewlett-Packard Proliant DL360p G8
  • Dual 6-core CPU, E5-2640 @ 2.5GHz
  • 16 x 8GB RDIMM @ 1333MHz (128GB)
  • 8 x 2.5″ 500GB HDD, SATA
  • SmartArray P420i 1GB Cache

Disk/RAID Configuration

All internal drives comprise a single RAID6 configuration with two logical volumes exposed to the OS.  da0 is a 100GB volume while da1 is comprised of the remaining available disk space.

da0 is reserved for the OS and user-land.  These experiments are executed on an empty da1 of approximately 2T in size.

Software

The following software was used in these experiments:

Assumptions

  • I/O is dominated by large sequential writes
  • Random I/O is negligible

Methodology

Testing of the I/O subsystem was performed utilizing iozone for an application independent perspective.   In the interest of minimizing the amount of time necessary to execute these experiments, we minimized the number of test cases to remove those of less value given the assumptions.

There were two primary tests; Writing of a 256GB file in conjunction with iozone’s -e and -o command-line options which causes each write to occur asynchronously w/ a full flush to disk.  This is intended to be a worst-case scenario eliminating as much cache/buffer benefits and forcing I/O to be fully absorbed by the disks and disk controller. Secondly, testing was completed on a 75MB file while not utilizing the -e or -o options to iozone.  This is intended to provide a best case scenario where the effects of OS level caching and buffering are realized.

A record size of 4096k was used as independent tests by a colleague showed this size providing the best performance.  In the various different configurations, a different size may have provided better results, but they were not tested in an effort to minimize the number of test scenarios.

The tests were executed in a series of 3 samples averaging the results across all samples.

Test Results

Non-Journaled/Baseline

# tunefs -p /filesystem
tunefs: POSIX.1e ACLs: (-a)                                   disabled
tunefs: NFSv4 ACLs: (-N)                                      disabled
tunefs: MAC multilabel: (-l)                                  disabled
tunefs: soft updates: (-n)                                    enabled
tunefs: soft update journaling: (-j)                          disabled
tunefs: gjournal: (-J)                                        disabled
tunefs: trim: (-t)                                            disabled
tunefs: maximum blocks per file in a cylinder group: (-e)     4096
tunefs: average file size: (-f)                               16384
tunefs: average number of files in a directory: (-s)          64
tunefs: minimum percentage of free space: (-m)                8%
tunefs: space to hold for metadata blocks: (-k)               6408
tunefs: optimization preference: (-o)                         time
tunefs: volume label: (-L)
Test Command Result
1  iozone -r 4096k -s 256g -R -o -e -t 1 -i 0 -i 1 -F /dump/BIGGY Initial Write:  233 MB/sec
Rewrite:           506 MB/sec
Read:                 531 MB/sec
Re-read:           531 MB/sec
2  iozone -r 4096k -s 256g -R -t 1 -i 0 -i 1 -F /dump/BIGGY Initial Write:  512 MB/sec
Rewrite:            515 MB/sec
Read:                 536 MB/sec
Re-read:          536 MB/sec
3  iozone -r 4096k -s 75m -R -o -e -t 1 -i 0 -i 1 -F /dump/BIGGY Initial Write: 283 MB/sec
Rewrite:           563 MB/sec
Read:                 3962 MB/sec
Re-read:          4037 MB/sec
4  iozone -r 4096k -s 75m -R -t 1 -i 0 -i 1 -F /dump/BIGGY Initial Write: 1859 MB/sec
Rewrite:           2278 MB/sec
Read:                4098 MB/sec
Re-read:          4026 MB/sec
5  iozone -r 4096k -s 75m -R -t 6 -i 0 -F F1 F2 F3 F4 F5 F6 Initial Write: 1353 MB/sec
Rewrite:          4952 MB/sec
6  iozone -r 4096k -s 75m -R -t 7 -i 0 -F F1 F2 F3 F4 F5 F6 F7 Initial Write: 1028 MB/sec
Rewrite:          4042 MB/sec

SU+J

# tunefs -p /filesystem
tunefs: POSIX.1e ACLs: (-a)                                   disabled
tunefs: NFSv4 ACLs: (-N)                                      disabled
tunefs: MAC multilabel: (-l)                                  disabled
tunefs: soft updates: (-n)                                    enabled
tunefs: soft update journaling: (-j)                          enabled
tunefs: gjournal: (-J)                                        disabled
tunefs: trim: (-t)                                            disabled
tunefs: maximum blocks per file in a cylinder group: (-e)     4096
tunefs: average file size: (-f)                               16384
tunefs: average number of files in a directory: (-s)          64
tunefs: minimum percentage of free space: (-m)                8%
tunefs: space to hold for metadata blocks: (-k)               6408
tunefs: optimization preference: (-o)                         time
tunefs: volume label: (-L)
Test Command Result
1  iozone -r 4096k -s 256g -R -o -e -t 1 -i 0 -i 1 -F /dump/BIGGY Initial Write: 236 MB/sec
Rewrite:          506 MB/sec
Read:                532 MB/sec
Re-read:         533 MB/sec
2  iozone -r 4096k -s 256g -R -t 1 -i 0 -i 1 -F /dump/BIGGY Initial Write: 508 MB/sec
Rewrite:           514 MB/sec
Read:                525 MB/sec
Re-read:         525 MB/sec
3  iozone -r 4096k -s 75m -R -o -e -t 1 -i 0 -i 1 -F /dump/BIGGY Initial Write: 280 MB/sec
Rewrite:           577 MB/sec
Read:                4263 MB/sec
Re-read:         4196 MB/sec
4  iozone -r 4096k -s 75m -R -t 1 -i 0 -i 1 -F /dump/BIGGY Initial Write: 1756 MB/sec
Rewrite:          2227 MB/sec
Read:               4201 MB/sec
Re-read:         4319 MB/sec
5  iozone -r 4096k -s 75m -R -t 6 -i 0 -F F1 F2 F3 F4 F5 F6 Initial Write: 1148 MB/sec
Rewrite:           4998 MB/sec
6  iozone -r 4096k -s 75m -R -t 7 -i 0 -F F1 F2 F3 F4 F5 F6 F7 Initial Write: 924 MB/sec
Rewrite:           4551 MB/sec

gjournal

# tunefs -p /filesystem
tunefs: POSIX.1e ACLs: (-a)                                   disabled
tunefs: NFSv4 ACLs: (-N)                                      disabled
tunefs: MAC multilabel: (-l)                                  disabled
tunefs: soft updates: (-n)                                    enabled
tunefs: soft update journaling: (-j)                          disabled
tunefs: gjournal: (-J)                                        enabled
tunefs: trim: (-t)                                            disabled
tunefs: maximum blocks per file in a cylinder group: (-e)     4096
tunefs: average file size: (-f)                               16384
tunefs: average number of files in a directory: (-s)          64
tunefs: minimum percentage of free space: (-m)                8%
tunefs: space to hold for metadata blocks: (-k)               6408
tunefs: optimization preference: (-o)                         time
tunefs: volume label: (-L)
Test Command Result
1  iozone -r 4096k -s 256g -R -o -e -t 1 -i 0 -i 1 -F /dump/BIGGY Initial Write: 176 MB/sec
Rewrite:          176 MB/sec
Read:               505 MB/sec
Re-read:         510 MB/sec
2  iozone -r 4096k -s 256g -R -t 1 -i 0 -i 1 -F /dump/BIGGY Initial Write:  181 MB/sec
Rewrite:           184 MB/sec
Read:                 507 MB/sec
Re-read:            509 MB/sec
3  iozone -r 4096k -s 75m -R -o -e -t 1 -i 0 -i 1 -F /dump/BIGGY Initial Write: 401 MB/sec
Rewrite:          492 MB/sec
Read:               4467 MB/sec
Re-read:         4087 MB/sec
4  iozone -r 4096k -s 75m -R -t 1 -i 0 -i 1 -F /dump/BIGGY Initial Write: 1627 MB/sec
Rewrite:          1415 MB/sec
Read:               4838 MB/sec
Re-read:        4879 MB/sec
5 iozone -r 4096k -s 75m -R -t 6 -i 0 -F F1 F2 F3 F4 F5 F6 Initial Write: 943 MB/sec
Rewrite:           872 MB/sec
6 iozone -r 4096k -s 75m -R -t 7 -i 0 -F F1 F2 F3 F4 F5 F6 F7 Initial Write: 374 MB/sec
Rewrite:           367 MB/sec

Disclaimer

Data and information described on this blog are for informational purposes only. The author of this blog provides no warranty/guarantee, expressed or implied, that this data and information will function as described here. Readers are expected to exercise due diligence when researching, developing, and deploying techniques and methods for use within their environments.

Comments posted are the explicit opinions of the comment poster themselves and does not necessarily reflect the views and opinions of the author of this blog.

Advertisements
Categories: FreeBSD
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: