/
Hotplug NVMe SSDs

Hotplug NVMe SSDs

Tips and instructions on how to replace a faulty NVMe SSD without having to bring down a DAOS system. This guide focuses on what to do on a production DAOS deployment to enable reintegration of excluded targets after SSD failure.

Reference: https://daosio.atlassian.net/browse/DAOS-17014

 Instructions

NOTE: enable_hotplug: true must be set in DAOS server config file prior to formatting DAOS server storage during installation and deployment in order to enable hotplug functionality.

Summary of procedure:

  • NVMe SSD in use by DAOS engine starts throwing media errors

  • Either device is automatically set faulty by DAOS (RAS events emitted and targets excluded from pools) or manually set device with media errors as faulty

  • Drive with media errors replaced with working SSD

  • “NEW” SMD device appears in dmg storage query list-devices output

  • dmg storage replace nvme... command used to swap old UUID from list-devices with new entry

  • Targets reintegrate automatically after short while (~2mins)

Steps:

  1. I/O errors seen on drive in slot 5

tanabarr@daos-0235:~> dmg storage scan --nvme-health -l daos-0235:10003 -i| grep 'PCI:\|Media' PCI:4a0005:81:00.0 Model:SAMSUNG MZWLR15THALA-00007 FW:MPK95B5Q Socket:0 Capacity:15 TB Media Errors:0 PCI:4a0005:83:00.0 Model:SAMSUNG MZWLR15THALA-00007 FW:MPK95B5Q Socket:0 Capacity:15 TB Media Errors:0 PCI:4a0005:85:00.0 Model:SAMSUNG MZWLR15THALA-00007 FW:MPK95B5Q Socket:0 Capacity:15 TB Media Errors:0 PCI:4a0005:87:00.0 Model:SAMSUNG MZWLR15THALA-00007 FW:MPK95B5Q Socket:0 Capacity:15 TB Media Errors:0 PCI:640005:81:00.0 Model:SAMSUNG MZWLR15THALA-00007 FW:MPK95B5Q Socket:0 Capacity:15 TB Media Errors:1943 PCI:640005:83:00.0 Model:SAMSUNG MZWLR15THALA-00007 FW:MPK95B5Q Socket:0 Capacity:15 TB Media Errors:0 PCI:640005:85:00.0 Model:SAMSUNG MZWLR15THALA-00007 FW:MPK95B5Q Socket:0 Capacity:15 TB Media Errors:0 PCI:640005:87:00.0 Model:SAMSUNG MZWLR15THALA-00007 FW:MPK95B5Q Socket:0 Capacity:15 TB Media Errors:0 PCI:970005:01:00.0 Model:SAMSUNG MZWLR15THALA-00007 FW:MPK95B5Q Socket:1 Capacity:15 TB Media Errors:0 PCI:970005:03:00.0 Model:SAMSUNG MZWLR15THALA-00007 FW:MPK95B5Q Socket:1 Capacity:15 TB Media Errors:0 PCI:970005:05:00.0 Model:SAMSUNG MZWLR15THALA-00007 FW:MPK95B5Q Socket:1 Capacity:15 TB Media Errors:0 PCI:970005:07:00.0 Model:SAMSUNG MZWLR15THALA-00007 FW:MPK95B5Q Socket:1 Capacity:15 TB Media Errors:0 PCI:e20005:01:00.0 Model:SAMSUNG MZWLR15THALA-00007 FW:MPK95B5Q Socket:1 Capacity:15 TB Media Errors:0 PCI:e20005:03:00.0 Model:SAMSUNG MZWLR15THALA-00007 FW:MPK95B5Q Socket:1 Capacity:15 TB Media Errors:0 PCI:e20005:05:00.0 Model:SAMSUNG MZWLR15THALA-00007 FW:MPK95B5Q Socket:1 Capacity:15 TB Media Errors:0 PCI:e20005:07:00.0 Model:SAMSUNG MZWLR15THALA-00007 FW:MPK95B5Q Socket:1 Capacity:15 TB Media Errors:0

  1. Single node cluster started and storage formatted, pool created

tanabarr@daos-0235:~> sudo systemctl start daos_server tanabarr@daos-0235:~> tail -f /tmp/daos_control.log DEBUG 2025/02/13 15:25:40.461861 instance_storage.go:126: instance 0: needsMetaFormat: false DEBUG 2025/02/13 15:25:40.461871 instance_storage.go:126: instance 1: needsMetaFormat: false DEBUG 2025/02/13 15:25:40.461881 provider.go:359: /mnt/daos0: checking formatting DEBUG 2025/02/13 15:25:40.461887 provider.go:359: /mnt/daos1: checking formatting DEBUG 2025/02/13 15:25:40.473986 provider.go:372: /mnt/daos1 (dcpm) needs format: true DEBUG 2025/02/13 15:25:40.474001 instance_storage.go:132: instance 1: needsScmFormat: true daos-0235 INFO 2025/02/13 15:25:40 SCM format required on instance 1 DEBUG 2025/02/13 15:25:40.474361 provider.go:372: /mnt/daos0 (dcpm) needs format: true DEBUG 2025/02/13 15:25:40.474377 instance_storage.go:132: instance 0: needsScmFormat: true daos-0235 INFO 2025/02/13 15:25:40 SCM format required on instance 0 ^C tanabarr@daos-0235:~> dmg storage format -l daos-0235:10003 -i Format Summary: Hosts SCM Devices NVMe Devices ----- ----------- ------------ daos-0235 2 16 tanabarr@daos-0235:~> dmg pool create tpool1 -z 100% -i Creating DAOS pool with 100% of all storage Pool created with 3.16%,96.84% storage tier ratio ------------------------------------------------- UUID : c66b7706-f07e-48e9-8470-be56b176bf74 Service Leader : 1 Service Ranks : [0-1] Storage Ranks : [0-1] Total Size : 254 TB Storage tier 0 (SCM) : 8.0 TB (4.0 TB / rank) Storage tier 1 (NVMe): 246 TB (123 TB / rank)

  1. Set NVMe drive faulty and verify device state

tanabarr@daos-0235:~> dmg storage set nvme-faulty -l daos-0235:10003 -i -u c6bbd4df-b75c-43b7-a257-f5d254fda3ee NOTICE: This command will permanently mark the device as unusable! Are you sure you want to continue? (yes/no) yes set-faulty operation performed successfully on the following host: daos-0235:10003 tanabarr@daos-0235:~> dmg storage query list-devices -l daos-0235:10003 -i ---------------- daos-0235 ---------------- Devices UUID:f730404a-2022-42c8-ae31-9ee6f09a9af4 [TrAddr:4a0005:81:00.0 NSID:1] Roles:NA Targets:[0 8] Rank:0 State:NORMAL LED:OFF UUID:f5a49c73-1d74-4b0b-8038-d489ca14c182 [TrAddr:4a0005:83:00.0 NSID:1] Roles:NA Targets:[1 9] Rank:0 State:NORMAL LED:OFF UUID:bc87775d-1823-4f5f-b09a-bf779cf66340 [TrAddr:4a0005:85:00.0 NSID:1] Roles:NA Targets:[2 10] Rank:0 State:NORMAL LED:OFF UUID:915fc41c-4d9a-46af-904a-ec094dd5f7df [TrAddr:4a0005:87:00.0 NSID:1] Roles:NA Targets:[3 11] Rank:0 State:NORMAL LED:OFF UUID:c6bbd4df-b75c-43b7-a257-f5d254fda3ee [TrAddr:640005:81:00.0 NSID:1] Roles:NA Targets:[4 12] Rank:0 State:EVICTED LED:ON UUID:98ff55d3-e512-4487-ad23-62efa35bc7ae [TrAddr:640005:83:00.0 NSID:1] Roles:NA Targets:[5 13] Rank:0 State:NORMAL LED:OFF UUID:c7983575-6d0b-45a6-b4a8-9ca2820c0785 [TrAddr:640005:85:00.0 NSID:1] Roles:NA Targets:[6 14] Rank:0 State:NORMAL LED:OFF UUID:fba7cd0e-17ef-424e-8007-0f885dbf27c4 [TrAddr:640005:87:00.0 NSID:1] Roles:NA Targets:[7 15] Rank:0 State:NORMAL LED:OFF UUID:cac76138-097f-4c60-8972-5760dcf1d755 [TrAddr:970005:01:00.0 NSID:1] Roles:NA Targets:[0 8] Rank:1 State:NORMAL LED:OFF UUID:b349f3af-758d-4dd5-a381-dd9d95c9079c [TrAddr:970005:03:00.0 NSID:1] Roles:NA Targets:[1 9] Rank:1 State:NORMAL LED:OFF UUID:4b0b556b-69e7-4cfb-aa55-81e99897a92c [TrAddr:970005:05:00.0 NSID:1] Roles:NA Targets:[2 10] Rank:1 State:NORMAL LED:OFF UUID:a635c73d-c2b8-4ca8-8c50-f57505c5f3eb [TrAddr:970005:07:00.0 NSID:1] Roles:NA Targets:[3 11] Rank:1 State:NORMAL LED:OFF UUID:f37fec23-73c6-4975-a908-75e22c9e0122 [TrAddr:e20005:01:00.0 NSID:1] Roles:NA Targets:[4 12] Rank:1 State:NORMAL LED:OFF UUID:97f15ced-e95a-4cfc-9fc8-7a5780511469 [TrAddr:e20005:03:00.0 NSID:1] Roles:NA Targets:[5 13] Rank:1 State:NORMAL LED:OFF UUID:35e579b5-606a-4878-ad93-2201f7e1822e [TrAddr:e20005:05:00.0 NSID:1] Roles:NA Targets:[6 14] Rank:1 State:NORMAL LED:OFF UUID:a1530968-475f-45ca-a6da-b1365eccde53 [TrAddr:e20/005:07:00.0 NSID:1] Roles:NA Targets:[7 15] Rank:1 State:NORMAL LED:OFF

  1. Perform hot swap, I/O errored drive for fresh known-working drive. On hot-remove, log activity indicates errors from removed drive,

    1. If VMD is not enabled, use dmg storage nvme-rebind command after hot-plug to bind the new NVMe SSD to VFIO userspace so it is usable by the daos_engine SPDK process. See https://docs.daos.io/v2.7/admin/administration/#exclusion-and-hotplug for a command usage reference.

Once the new devices visible to the engine processes, “NEW” and “UNPLUGGED” device entries appear in dmg storage query list-devices output.

daos-0235 ERROR 2025/02/13 15:43:57 daos_engine:0 [2025-02-13 15:43:57.669465] nvme_pcie_common.c: 766:nvme_pcie_qpair_abort_trackers: *ERROR*: aborting outstanding command [2025-02-13 15:43:57.669497] nvme_pcie_common.c: 766:nvme_pcie_qpair_abort_trackers: *ERROR*: aborting outstanding command [2025-02-13 15:43:57.669509] nvme_pcie_common.c: 766:nvme_pcie_qpair_abort_trackers: *ERROR*: aborting outstanding command [2025-02-13 15:43:57.669517] nvme_pcie_common.c: 766:nvme_pcie_qpair_abort_trackers: *ERROR*: aborting outstanding command [2025-02-13 15:43:57.669524] nvme_pcie_common.c: 766:nvme_pcie_qpair_abort_trackers: *ERROR*: aborting outstanding command [2025-02-13 15:43:57.669530] nvme_pcie_common.c: 766:nvme_pcie_qpair_abort_trackers: *ERROR*: aborting outstanding command [2025-02-13 15:43:57.669536] nvme_pcie_common.c: 766:nvme_pcie_qpair_abort_trackers: *ERROR*: aborting outstanding command [2025-02-13 15:43:57.669543] nvme_pcie_common.c: 766:nvme_pcie_qpair_abort_trackers: *ERROR*: aborting outstanding command [2025-02-13 15:43:57.669549] nvme_ctrlr.c:1052:nvme_ctrlr_fail: *ERROR*: [640005:81:00.0] in failed state. TRACE 2025/02/13 15:43:57.670520 mgmt_drpc.go:88: srv upcall: cluster event: success daos-0235 ERROR 2025/02/13 15:44:12 daos_engine:0 [2025-02-13 15:44:12.785916] nvme_ctrlr.c:5170:spdk_nvme_ctrlr_security_receive: *ERROR*: [640005:81:00.0] spdk_nvme_ctrlr_cmd_security_receive failed! daos-0235 INFO 2025/02/13 15:45:10 daos_engine:0 Detected hot plugged device HotInNvme0n1 TRACE 2025/02/13 15:45:10.805767 mgmt_drpc.go:88: srv upcall: cluster event: success tanabarr@daos-0235:~> dmg storage query list-devices -l daos-0235:10003 -i ---------------- daos-0235 ---------------- Devices UUID:f730404a-2022-42c8-ae31-9ee6f09a9af4 [TrAddr:4a0005:81:00.0 NSID:1] Roles:NA Targets:[0 8] Rank:0 State:NORMAL LED:OFF UUID:f5a49c73-1d74-4b0b-8038-d489ca14c182 [TrAddr:4a0005:83:00.0 NSID:1] Roles:NA Targets:[1 9] Rank:0 State:NORMAL LED:OFF UUID:bc87775d-1823-4f5f-b09a-bf779cf66340 [TrAddr:4a0005:85:00.0 NSID:1] Roles:NA Targets:[2 10] Rank:0 State:NORMAL LED:OFF UUID:915fc41c-4d9a-46af-904a-ec094dd5f7df [TrAddr:4a0005:87:00.0 NSID:1] Roles:NA Targets:[3 11] Rank:0 State:NORMAL LED:OFF UUID:c6bbd4df-b75c-43b7-a257-f5d254fda3ee [TrAddr:] Roles:NA Targets:[4 12] Rank:0 State:UNPLUGGED LED:NA UUID:98ff55d3-e512-4487-ad23-62efa35bc7ae [TrAddr:640005:83:00.0 NSID:1] Roles:NA Targets:[5 13] Rank:0 State:NORMAL LED:OFF UUID:c7983575-6d0b-45a6-b4a8-9ca2820c0785 [TrAddr:640005:85:00.0 NSID:1] Roles:NA Targets:[6 14] Rank:0 State:NORMAL LED:OFF UUID:fba7cd0e-17ef-424e-8007-0f885dbf27c4 [TrAddr:640005:87:00.0 NSID:1] Roles:NA Targets:[7 15] Rank:0 State:NORMAL LED:OFF UUID:2fb8751d-859a-4742-8eea-740726237c83 [TrAddr:640005:81:00.0 NSID:1] Roles:NA Targets:[] Rank:0 State:NEW LED:ON UUID:cac76138-097f-4c60-8972-5760dcf1d755 [TrAddr:970005:01:00.0 NSID:1] Roles:NA Targets:[0 8] Rank:1 State:NORMAL LED:OFF UUID:b349f3af-758d-4dd5-a381-dd9d95c9079c [TrAddr:970005:03:00.0 NSID:1] Roles:NA Targets:[1 9] Rank:1 State:NORMAL LED:OFF UUID:4b0b556b-69e7-4cfb-aa55-81e99897a92c [TrAddr:970005:05:00.0 NSID:1] Roles:NA Targets:[2 10] Rank:1 State:NORMAL LED:OFF UUID:a635c73d-c2b8-4ca8-8c50-f57505c5f3eb [TrAddr:970005:07:00.0 NSID:1] Roles:NA Targets:[3 11] Rank:1 State:NORMAL LED:OFF UUID:f37fec23-73c6-4975-a908-75e22c9e0122 [TrAddr:e20005:01:00.0 NSID:1] Roles:NA Targets:[4 12] Rank:1 State:NORMAL LED:OFF UUID:97f15ced-e95a-4cfc-9fc8-7a5780511469 [TrAddr:e20005:03:00.0 NSID:1] Roles:NA Targets:[5 13] Rank:1 State:NORMAL LED:OFF UUID:35e579b5-606a-4878-ad93-2201f7e1822e [TrAddr:e20005:05:00.0 NSID:1] Roles:NA Targets:[6 14] Rank:1 State:NORMAL LED:OFF UUID:a1530968-475f-45ca-a6da-b1365eccde53 [TrAddr:e20005:07:00.0 NSID:1] Roles:NA Targets:[7 15] Rank:1 State:NORMAL LED:OFF

  1. Run replace command to swap the unplugged device entry to the new device entry, verify normal state after replace

tanabarr@daos-0235:~> dmg storage replace nvme --old-uuid c6bbd4df-b75c-43b7-a257-f5d254fda3ee --new-uuid 2fb8751d-859a-4742-8eea-740726237c83 -l daos-0235:10003 dev-replace operation performed successfully on the following host: daos-0235:10003 tanabarr@daos-0235:~> dmg storage query list-devices -l daos-0235:10003 -i ---------------- daos-0235 ---------------- Devices UUID:f730404a-2022-42c8-ae31-9ee6f09a9af4 [TrAddr:4a0005:81:00.0 NSID:1] Roles:NA Targets:[0 8] Rank:0 State:NORMAL LED:OFF UUID:f5a49c73-1d74-4b0b-8038-d489ca14c182 [TrAddr:4a0005:83:00.0 NSID:1] Roles:NA Targets:[1 9] Rank:0 State:NORMAL LED:OFF UUID:bc87775d-1823-4f5f-b09a-bf779cf66340 [TrAddr:4a0005:85:00.0 NSID:1] Roles:NA Targets:[2 10] Rank:0 State:NORMAL LED:OFF UUID:915fc41c-4d9a-46af-904a-ec094dd5f7df [TrAddr:4a0005:87:00.0 NSID:1] Roles:NA Targets:[3 11] Rank:0 State:NORMAL LED:OFF UUID:98ff55d3-e512-4487-ad23-62efa35bc7ae [TrAddr:640005:83:00.0 NSID:1] Roles:NA Targets:[5 13] Rank:0 State:NORMAL LED:OFF UUID:c7983575-6d0b-45a6-b4a8-9ca2820c0785 [TrAddr:640005:85:00.0 NSID:1] Roles:NA Targets:[6 14] Rank:0 State:NORMAL LED:OFF UUID:fba7cd0e-17ef-424e-8007-0f885dbf27c4 [TrAddr:640005:87:00.0 NSID:1] Roles:NA Targets:[7 15] Rank:0 State:NORMAL LED:OFF UUID:2fb8751d-859a-4742-8eea-740726237c83 [TrAddr:640005:81:00.0 NSID:1] Roles:NA Targets:[4 12] Rank:0 State:NORMAL LED:ON UUID:cac76138-097f-4c60-8972-5760dcf1d755 [TrAddr:970005:01:00.0 NSID:1] Roles:NA Targets:[0 8] Rank:1 State:NORMAL LED:OFF UUID:b349f3af-758d-4dd5-a381-dd9d95c9079c [TrAddr:970005:03:00.0 NSID:1] Roles:NA Targets:[1 9] Rank:1 State:NORMAL LED:OFF UUID:4b0b556b-69e7-4cfb-aa55-81e99897a92c [TrAddr:970005:05:00.0 NSID:1] Roles:NA Targets:[2 10] Rank:1 State:NORMAL LED:OFF UUID:a635c73d-c2b8-4ca8-8c50-f57505c5f3eb [TrAddr:970005:07:00.0 NSID:1] Roles:NA Targets:[3 11] Rank:1 State:NORMAL LED:OFF UUID:f37fec23-73c6-4975-a908-75e22c9e0122 [TrAddr:e20005:01:00.0 NSID:1] Roles:NA Targets:[4 12] Rank:1 State:NORMAL LED:OFF UUID:97f15ced-e95a-4cfc-9fc8-7a5780511469 [TrAddr:e20005:03:00.0 NSID:1] Roles:NA Targets:[5 13] Rank:1 State:NORMAL LED:OFF UUID:35e579b5-606a-4878-ad93-2201f7e1822e [TrAddr:e20005:05:00.0 NSID:1] Roles:NA Targets:[6 14] Rank:1 State:NORMAL LED:OFF UUID:a1530968-475f-45ca-a6da-b1365eccde53 [TrAddr:e20005:07:00.0 NSID:1] Roles:NA Targets:[7 15] Rank:1 State:NORMAL LED:OFF tanabarr@daos-0235:~> dmg pool query tpool1 Pool c66b7706-f07e-48e9-8470-be56b176bf74, ntarget=32, disabled=2, leader=1, version=5, state=Degraded Pool health info: - Rebuild done, 0 objs, 0 recs Pool space info: - Target(VOS) count:30 - Storage tier 0 (SCM): Total size: 7.5 TB Free: 7.2 TB, min:239 GB, max:239 GB, mean:239 GB - Storage tier 1 (NVMe): Total size: 230 TB Free: 230 TB, min:7.7 TB, max:7.7 TB, mean:7.7 TB <~2min wait> tanabarr@daos-0235:~> dmg pool query tpool1 Pool c66b7706-f07e-48e9-8470-be56b176bf74, ntarget=32, disabled=0, leader=1, version=9, state=Ready Pool health info: - Rebuild done, 0 objs, 0 recs Pool space info: - Target(VOS) count:32 - Storage tier 0 (SCM): Total size: 8.0 TB Free: 7.7 TB, min:239 GB, max:239 GB, mean:239 GB - Storage tier 1 (NVMe): Total size: 246 TB Free: 246 TB, min:7.7 TB, max:7.7 TB, mean:7.7 TB

NOTE: Pool query output took a little time to recognise the reintegration of the targets which was performed automatically after dmg storage replace nvme command.

  1. Then reset LED state, this should happen automatically see https://daosio.atlassian.net/browse/DAOS-16999

tanabarr@daos-0235:~> dmg storage led identify --reset 640005:81:00.0 -l daos-0235 -i ---------------- daos-0235 ---------------- Devices TrAddr:640005:81:00.0 LED:OFF tanabarr@daos-0235:~> dmg storage led check 640005:81:00.0 -l localhost -i --------- localhost --------- Devices TrAddr:640005:81:00.0 LED:OFF

 Related articles