So I just rebuilt my little home server RAID from LVM+Ext4 to ZFS, changing the layout from RAID5 to RAID1+0, consisting of a pool of two mirrored disk sets.
Since I’m a cheap bugger frugal, I still run a small HP MicroServer Gen7 (N54L) with only 2 GB of RAM, which I’ve filled up with 4 x 3 TB WD RED drives for storage, and a 60 GB SSD for the system.
As everybody knows, the only difference between screwing around and science is writing it down. I was slightly too eager to get started to remember to benchmark my old file system, so I guess any future changes will have to be compared to the current set-up. Anyhow, here’s the baseline:
So, as a slightly less bright-sounding person asked: What does it all mean?
I’ve measured pure sequential reads, pure sequential writes, and how many random disk operations I can perform per second – split 80/20 into reads and writes – with 1, 4, 8, and 16 worker processes. All runs were performed with a 4 KB block size. The numbers speak for themselves, but I thought the yellow line representing the I/O capability looked a bit misleading, so I added the orange line as the quotient of how many I/O operations each worker process was able to generate per second.
The first main point to take away from this is that my writes don’t seem very fast. I’m pretty sure that adding some memory to the server would help quite a bit here, and there’s probably a good reason for why ZFS supports ZIL and SLOG devices.
The second point is relevant if – as I do – I would like to use a separate file server like this one for storing virtual machines in the future. VMware recommends a minimum capacity of about 30 IOPS for an application server. The orange line indicates that I definitely should run less than eight such machines from this file server.
About the benchmarks
I used a program called fio and wrote a configuration file for it that looks something like this:
[global] rw=read size=8G io_limit=8G directory=/export/temp rwmixread=80 numjobs=16 runtime=5m [perftest]
The colored fields were changed between runs. The rw parameter was set to read for sequential reads, write for sequential writes (surprise!), and randrw for random reads and writes.
The size was set to four times the RAM of the server, to make it harder for the server to fake results by presenting cached data.
The io_limit was set to avoid having to wait while all 8 GB files were fully written to. In case of the IOPS tests, I instead used the “runtime” parameter to limit the amount of time each test ran.
Numjobs was set to 1, 4, 8 or 16, for each of the types of load I tested.