In order to evaluate the performance impact of phase 1 on DAOS, a set of low-level benchmark using spdk_nvme_perf can be used to (try to) emulate the WAL traffic. This wiki page summarizes all the command line option as well as the results.
To know what type of SSD we are dealing with, the spdk_nvme_identify command can be used:
# spdk_nvme_identify -V -r 'trtype:PCIe traddr:e20005:04:00.0' EAL: No free 2048 kB hugepages reported on node 1 EAL: No available 1048576 kB hugepages reported TELEMETRY: No legacy callbacks, legacy socket not created ===================================================== NVMe Controller at e20005:04:00.0 [144d:a824] ===================================================== Controller Capabilities/Features ================================ Vendor ID: 144d Subsystem Vendor ID: 144d Serial Number: S4YPNE0N800120 Model Number: SAMSUNG MZWLJ3T8HBLS-00007 Firmware Version: EPK98B5Q [...] |
It dumps a lot of information, but what we are mostly interested here is the model number, firmware version and heath information. For the latter, we are expecting results to be reported with relatively new SSD. This includes no critical warnings, “Available Spare” to 100% and “Life Percentage Used” to 0%.
The intent here is to measure that extra latency that the WAL commit might add to a single update operation. To do so, we want to first benchmark with a queue depth of 1 and from a single thread.
# spdk_nvme_perf -V -r 'trtype:PCIe traddr:e20005:04:00.0' -q 1 -o 4096 -w write -c 0x1 -t 60
EAL: No free 2048 kB hugepages reported on node 1
EAL: No available 1048576 kB hugepages reported
TELEMETRY: No legacy callbacks, legacy socket not created
Initializing NVMe Controllers
Attached to NVMe Controller at e20005:04:00.0 [144d:a824]
Associating PCIE (e20005:04:00.0) NSID 1 with lcore 0
Initialization complete. Launching workers.
========================================================
Latency(us)
Device Information : IOPS MiB/s Average min max
PCIE (e20005:04:00.0) NSID 1 from core 0: 102351.15 399.81 9.76 8.55 49644.32
========================================================
Total : 102351.15 399.81 9.76 8.55 49644.32 |
The result of interest here is the average latency and IOPS. With this SSD (PM1733), the average latency is 9.7us and 102K IOPS.
In practice, we are going to commit multiple operations in parallel to the WAL, so the latency with a queue depth > 1 is of interest. To see how the latency evolve, we would like to measure the latency for qd = 2^n with n in 1..8
# for i in `seq 8`; do echo $i; spdk_nvme_perf -V -r 'trtype:PCIe traddr:e20005:04:00.0' -q $((2**$i)) -o 4096 -w write -c 0x1 -t 60; done
[...]
Latency(us)
Device Information : IOPS MiB/s Average min max
PCIE (e20005:04:00.0) NSID 1 from core 0: 195578.43 763.98 10.22 8.52 65762.35
========================================================
Total : 195578.43 763.98 10.22 8.52 65762.35
[...]
Latency(us)
Device Information : IOPS MiB/s Average min max
PCIE (e20005:04:00.0) NSID 1 from core 0: 332707.70 1299.64 12.01 8.51 4215.47
========================================================
Total : 332707.70 1299.64 12.01 8.51 4215.47
[...]
Latency(us)
Device Information : IOPS MiB/s Average min max
PCIE (e20005:04:00.0) NSID 1 from core 0: 494580.12 1931.95 16.16 8.56 1273.60
========================================================
Total : 494580.12 1931.95 16.16 8.56 1273.60
[...]
Latency(us)
Device Information : IOPS MiB/s Average min max
PCIE (e20005:04:00.0) NSID 1 from core 0: 604705.96 2362.13 26.44 8.79 907.45
========================================================
Total : 604705.96 2362.13 26.44 8.79 907.45
[...]
Latency(us)
Device Information : IOPS MiB/s Average min max
PCIE (e20005:04:00.0) NSID 1 from core 0: 620312.92 2423.10 51.56 9.27 786.01
========================================================
Total : 620312.92 2423.10 51.56 9.27 786.01
[...]
Latency(us)
Device Information : IOPS MiB/s Average min max
PCIE (e20005:04:00.0) NSID 1 from core 0: 630753.78 2463.88 101.45 9.46 4472.04
========================================================
Total : 630753.78 2463.88 101.45 9.46 4472.04
[...]
Latency(us)
Device Information : IOPS MiB/s Average min max
PCIE (e20005:04:00.0) NSID 1 from core 0: 632426.65 2470.42 202.37 9.31 3191.75
========================================================
Total : 632426.65 2470.42 202.37 9.31 3191.75
[...]
Latency(us)
Device Information : IOPS MiB/s Average min max
PCIE (e20005:04:00.0) NSID 1 from core 0: 633949.05 2476.36 403.80 9.82 5846.45
========================================================
Total : 633949.05 2476.36 403.80 9.82 5846.45 |
In practice, there will be multi-target per SSD, so it is important to measure the IOPS and latency with multiple concurrent threads (i.e. 2/4/8).
# for t in 3 f ff; do for i in `seq 0 8`; do echo $t $i; spdk_nvme_perf -V -r 'trtype:PCIe traddr:e20005:04:00.0' -q $((2**$i)) -o 4096 -w write -c 0x$t -t 60; done; done [...] |
To provide results, the following spreadsheet should be filed with the output of this command that collects all the results:
# for t in 1 3 f ff; do for i in `seq 0 8`; do echo $t $i; spdk_nvme_perf -V -r 'trtype:PCIe traddr:e20005:04:00.0' -q $((2**$i)) -o 4096 -w write -c 0x$t -t 60; done; done [...] |
Data should be filled in the format of the example spreadsheet below: