NVMe Test Plan

Use the default option for now.

[Nvme]
TransportID "trtype:PCIe traddr:0000:81:00.0" Nvme0
TransportID "trtype:PCIe traddr:0000:82:00.0" Nvme1
TimeoutUsec 0
ActionOnTimeout None
AdminPollRate 100000
HotplugEnable No
HotplugPollRate 0

src/vos/vea/tests/vea_ut

src/eio/smd/tests/smd_ut

Look at the DAOS-1246 and include all unit tests.

daos_epoch_discard() will also create the more fragmentation. So add unit test specific to that API call.

Write/Read data (<4K) 1B/1K/4K [Random/Sequential] and make sure it does not use NVMe

Write/Read data with non standard sizes 4025B,259K,1.1M,30.22M [Random/Sequential] and make sure it use NVMe

Write/Read data (>4K) 1M/16G [Random/Sequential] and make sure it use NVMe

Write/Read data with non standard sizes 4025,259K,1.1M,30.22M [Random/Sequential] and make sure it use NVMe

Write/Read data (<4K) 1B/1K/4K [Random/Sequential] and make sure it does not use NVMe

Write/Read data with non standard sizes 4025,259K,1.1M,30.22M [Random/Sequential] and make sure it use NVMe

For second case, data sizes can be generated random instead of predefined fixed size.

Use the maximum pool size based on NVMe size on each server and number of total servers.

Write/Read data (>4K) 1M/16G/1TB [Random/Sequential] and make sure it use NVMe

Write/Read data with non standard sizes 4025,259K,1.1M,30.22M [Random/Sequential] and make sure it use NVMe

Try using the offset from API or use the core Python API to modify the existing Array and read through.

Write 1M, modified 1bytes in different offsets. read through.

The test code daos_run_io_conf.c will be doing similar thing so worth to use same.

Write the data/ Reboot the node/ Start servers/read data back and check data integrity.

After server start SPDK need to be setup again.

Write the data/ Stop Server/Start Server/Read data back and check data integrity.

Use the maximum pool size based on NVMe size on each server and number of total servers.

Write the data/ Reboot the node/ Start server/read data back and check data integrity.

After server start SPDK need to be setup again.

Write single IOR data set

Read single IOR data set

Kill the server while IO is doing Write and start the server after 5-10 min. IO should continue after server start? Do the same when read is in progress.

Use the maximum pool size based on NVMe size on each server and number of total servers.

Write the data on NVMe >4K

Re-write using the same array with small size ~1-2 bytes which will go through SCM.

Do this and change ~100 bytes with the different data

Do fetch which will combined the record and verify it.

When overwriting the data will be kind of new epoch entry getting created. But it will use the old data set and update the new bytes value only.

During fetch Epoch will aggregated and provide the result with modified bytes.

Write the data on SCM <4K

Extend the data set using the same array with larger size >8K which will go through NVMe

Do this and repeat for few times with the different data

Fetch the data-set which will combined the records, verify it.

unit tests

PMDK may have some API to get the allocated but it wont be easy to use as it is. There is Pool Query which will provide in future which can be track the size to verify memory leak.

Example:

1> Create 3G pool

2> Write 1G data and before it goes for transaction commit shutdown the server.

3> Next when server start verify it should claim the full 3G space.

We need fault injection layer to control the PMDK transaction with server start/stop. Which can do server stop before doing PMDK transaction

Create the pool with 12GB.

Add 8GB size of data on NVMe and 4G on SCM. IO has to be written with different container (~100) with different sizes.

use more smaller sizes for NVMe (4K-32K) and for SCM (8Bytes-4K) to have more fragmentation.

Fragmentation information will be needed from pool query in future to validate fragmentation number. 

Shutdown the single server or Eject and make sure the data is getting rebuilt on another NVMe drive.

Verify the data integrity after NVMe rebuild.

SPDK can be used to measure the raw performance of the drive.

daos_perf VOS raw performance

daos_perf Echo mode

daos_perf DAOS mode.

Sizes To be covered for Performance: 4K,16K,1M,8M,32M.

Performance measurement for Read/Write and IOPS.

It's better to get all this different number in single graph/text file for different sizes for better comparison.

Create pool, update to fill the pool with many container, Destroy the container. Different IOR with different sizes can be run in parallel.

Run single IOR job to validate the performance number in-between.

This need to be done for hours/days (Fill the container delete the container).

Measure the performance throughout and at the end to validate it's not dropped at any point.

We need to measure performance degradation over time. This will be a good exercise to verify whether our TRIM support is efficient (require 3DNAND)

1> Create the small data set with Epoch1, 2, 3

2> Discard 3rd Epoch

3> Run single IOR to verify the performance.

Do the above steps for ~10000 times. Make sure at the end the discarded fragmentation is not creating the issue for new write and even performing up to the level for same IOR run in-between.

Use smaller sizes for NVMe (4K-32K) and for SCM (8Bytes-4K) to have more fragmentation.

Create the pool of small size (NVMe size 1G)

We need to create the small number of files using IOR.

Direct API can be used but use the different Epoch for each record (Check the DAOS_MD_CAP environment variable This can be setup low for testing purpose.)

Validate lots of small I/Os followed by a lot of metadata operations

(1)test_metadata_fillup: Test to verify no IO happens after metadata is full

(2)test_metadata_addremove: Verify metadata release the space after container delete.

(3)test_metadata_server_restart: Verify 2000 IOR small size container after server restart. Test will write IOR in 5 different threads for faster execution time. Each thread will create 400 (8bytes) containers to
the same pool. Restart the servers, read IOR container file written previously and validate data integrity by using IOR option "-R -G 1".

server/metadata.py

tags: pr,hw,small,

tags=server,metadata,metadata_fillup

metadata_addremove

metadata_ior

Verify container can be successfully deleted when the storage pool is full ACL grant/remove modification

(4)test_container_removal_after_der_nospace: Verify container can be successfully deleted when the storage pool is full ACL grant/remove modification.

server/metadata.py

tags: pr,hw,small,

tags=server,metadata, metadata_der_nospace

• NVMe SSD discovery with "discover" bindings
• NVMe SSD burn-in with "burnin" bindings
• NVMe SSD configuration with  "configuration" bindings

• evicting SSD based on high temperature
• wear-leveling data

• SSD firmware image update