Archive
Disk I/O Performance Under Filesystem Journaling on FreeBSD 10.3
Overview
This post describes disk I/O performance capabilities of the HP DL360p G8 in terms of filesystem journaling mechanisms on FreeBSD 10.3.
References
- https://people.freebsd.org/~kris/scaling/Help_my_system_is_slow.pdf
- http://docplayer.net/2690085-Hp-smart-array-controllers-and-basic-raid-performance-factors.html
- https://dom.as/2009/03/11/iostat/
Hardware
- Hewlett-Packard Proliant DL360p G8
- Dual 6-core CPU, E5-2640 @ 2.5GHz
- 16 x 8GB RDIMM @ 1333MHz (128GB)
- 8 x 2.5″ 500GB HDD, SATA
- SmartArray P420i 1GB Cache
Disk/RAID Configuration
All internal drives comprise a single RAID6 configuration with two logical volumes exposed to the OS. da0 is a 100GB volume while da1 is comprised of the remaining available disk space.
da0 is reserved for the OS and user-land. These experiments are executed on an empty da1 of approximately 2T in size.
Software
The following software was used in these experiments:
- FreeBSD 10.3
- iozone 3.457
- FreeBSD base iostat
- FreeBSD base gstat
Assumptions
- I/O is dominated by large sequential writes
- Random I/O is negligible
Methodology
Testing of the I/O subsystem was performed utilizing iozone for an application independent perspective. In the interest of minimizing the amount of time necessary to execute these experiments, we minimized the number of test cases to remove those of less value given the assumptions.
There were two primary tests; Writing of a 256GB file in conjunction with iozone’s -e and -o command-line options which causes each write to occur asynchronously w/ a full flush to disk. This is intended to be a worst-case scenario eliminating as much cache/buffer benefits and forcing I/O to be fully absorbed by the disks and disk controller. Secondly, testing was completed on a 75MB file while not utilizing the -e or -o options to iozone. This is intended to provide a best case scenario where the effects of OS level caching and buffering are realized.
A record size of 4096k was used as independent tests by a colleague showed this size providing the best performance. In the various different configurations, a different size may have provided better results, but they were not tested in an effort to minimize the number of test scenarios.
The tests were executed in a series of 3 samples averaging the results across all samples.
Test Results
Non-Journaled/Baseline
# tunefs -p /filesystem tunefs: POSIX.1e ACLs: (-a) disabled tunefs: NFSv4 ACLs: (-N) disabled tunefs: MAC multilabel: (-l) disabled tunefs: soft updates: (-n) enabled tunefs: soft update journaling: (-j) disabled tunefs: gjournal: (-J) disabled tunefs: trim: (-t) disabled tunefs: maximum blocks per file in a cylinder group: (-e) 4096 tunefs: average file size: (-f) 16384 tunefs: average number of files in a directory: (-s) 64 tunefs: minimum percentage of free space: (-m) 8% tunefs: space to hold for metadata blocks: (-k) 6408 tunefs: optimization preference: (-o) time tunefs: volume label: (-L)
| Test | Command | Result |
|---|---|---|
| 1 | iozone -r 4096k -s 256g -R -o -e -t 1 -i 0 -i 1 -F /dump/BIGGY | Initial Write: 233 MB/sec Rewrite: 506 MB/sec Read: 531 MB/sec Re-read: 531 MB/sec |
| 2 | iozone -r 4096k -s 256g -R -t 1 -i 0 -i 1 -F /dump/BIGGY | Initial Write: 512 MB/sec Rewrite: 515 MB/sec Read: 536 MB/sec Re-read: 536 MB/sec |
| 3 | iozone -r 4096k -s 75m -R -o -e -t 1 -i 0 -i 1 -F /dump/BIGGY | Initial Write: 283 MB/sec Rewrite: 563 MB/sec Read: 3962 MB/sec Re-read: 4037 MB/sec |
| 4 | iozone -r 4096k -s 75m -R -t 1 -i 0 -i 1 -F /dump/BIGGY | Initial Write: 1859 MB/sec Rewrite: 2278 MB/sec Read: 4098 MB/sec Re-read: 4026 MB/sec |
| 5 | iozone -r 4096k -s 75m -R -t 6 -i 0 -F F1 F2 F3 F4 F5 F6 | Initial Write: 1353 MB/sec Rewrite: 4952 MB/sec |
| 6 | iozone -r 4096k -s 75m -R -t 7 -i 0 -F F1 F2 F3 F4 F5 F6 F7 | Initial Write: 1028 MB/sec Rewrite: 4042 MB/sec |
SU+J
# tunefs -p /filesystem tunefs: POSIX.1e ACLs: (-a) disabled tunefs: NFSv4 ACLs: (-N) disabled tunefs: MAC multilabel: (-l) disabled tunefs: soft updates: (-n) enabled tunefs: soft update journaling: (-j) enabled tunefs: gjournal: (-J) disabled tunefs: trim: (-t) disabled tunefs: maximum blocks per file in a cylinder group: (-e) 4096 tunefs: average file size: (-f) 16384 tunefs: average number of files in a directory: (-s) 64 tunefs: minimum percentage of free space: (-m) 8% tunefs: space to hold for metadata blocks: (-k) 6408 tunefs: optimization preference: (-o) time tunefs: volume label: (-L)
| Test | Command | Result |
|---|---|---|
| 1 | iozone -r 4096k -s 256g -R -o -e -t 1 -i 0 -i 1 -F /dump/BIGGY | Initial Write: 236 MB/sec Rewrite: 506 MB/sec Read: 532 MB/sec Re-read: 533 MB/sec |
| 2 | iozone -r 4096k -s 256g -R -t 1 -i 0 -i 1 -F /dump/BIGGY | Initial Write: 508 MB/sec Rewrite: 514 MB/sec Read: 525 MB/sec Re-read: 525 MB/sec |
| 3 | iozone -r 4096k -s 75m -R -o -e -t 1 -i 0 -i 1 -F /dump/BIGGY | Initial Write: 280 MB/sec Rewrite: 577 MB/sec Read: 4263 MB/sec Re-read: 4196 MB/sec |
| 4 | iozone -r 4096k -s 75m -R -t 1 -i 0 -i 1 -F /dump/BIGGY | Initial Write: 1756 MB/sec Rewrite: 2227 MB/sec Read: 4201 MB/sec Re-read: 4319 MB/sec |
| 5 | iozone -r 4096k -s 75m -R -t 6 -i 0 -F F1 F2 F3 F4 F5 F6 | Initial Write: 1148 MB/sec Rewrite: 4998 MB/sec |
| 6 | iozone -r 4096k -s 75m -R -t 7 -i 0 -F F1 F2 F3 F4 F5 F6 F7 | Initial Write: 924 MB/sec Rewrite: 4551 MB/sec |
gjournal
# tunefs -p /filesystem tunefs: POSIX.1e ACLs: (-a) disabled tunefs: NFSv4 ACLs: (-N) disabled tunefs: MAC multilabel: (-l) disabled tunefs: soft updates: (-n) enabled tunefs: soft update journaling: (-j) disabled tunefs: gjournal: (-J) enabled tunefs: trim: (-t) disabled tunefs: maximum blocks per file in a cylinder group: (-e) 4096 tunefs: average file size: (-f) 16384 tunefs: average number of files in a directory: (-s) 64 tunefs: minimum percentage of free space: (-m) 8% tunefs: space to hold for metadata blocks: (-k) 6408 tunefs: optimization preference: (-o) time tunefs: volume label: (-L)
| Test | Command | Result |
|---|---|---|
| 1 | iozone -r 4096k -s 256g -R -o -e -t 1 -i 0 -i 1 -F /dump/BIGGY | Initial Write: 176 MB/sec Rewrite: 176 MB/sec Read: 505 MB/sec Re-read: 510 MB/sec |
| 2 | iozone -r 4096k -s 256g -R -t 1 -i 0 -i 1 -F /dump/BIGGY | Initial Write: 181 MB/sec Rewrite: 184 MB/sec Read: 507 MB/sec Re-read: 509 MB/sec |
| 3 | iozone -r 4096k -s 75m -R -o -e -t 1 -i 0 -i 1 -F /dump/BIGGY | Initial Write: 401 MB/sec Rewrite: 492 MB/sec Read: 4467 MB/sec Re-read: 4087 MB/sec |
| 4 | iozone -r 4096k -s 75m -R -t 1 -i 0 -i 1 -F /dump/BIGGY | Initial Write: 1627 MB/sec Rewrite: 1415 MB/sec Read: 4838 MB/sec Re-read: 4879 MB/sec |
| 5 | iozone -r 4096k -s 75m -R -t 6 -i 0 -F F1 F2 F3 F4 F5 F6 | Initial Write: 943 MB/sec Rewrite: 872 MB/sec |
| 6 | iozone -r 4096k -s 75m -R -t 7 -i 0 -F F1 F2 F3 F4 F5 F6 F7 | Initial Write: 374 MB/sec Rewrite: 367 MB/sec |
Disclaimer
Data and information described on this blog are for informational purposes only. The author of this blog provides no warranty/guarantee, expressed or implied, that this data and information will function as described here. Readers are expected to exercise due diligence when researching, developing, and deploying techniques and methods for use within their environments.
Comments posted are the explicit opinions of the comment poster themselves and does not necessarily reflect the views and opinions of the author of this blog.
hostileadmin 