Server configuration file details
- Maureen Jean
- Phillip Henderson
Owned by Maureen Jean
Define the server name by uncommenting the following line
# ## Name associated with the DAOS system. ## Immutable after reformat. # ## NOTE: Changing the name is not supported in DAOS 1.0, it must not be changed ## from the default "daos_server". # ## default: daos_server name: daos_server
Define the access points for the server by uncommenting the following line. This list should contain the first node name currently defined in $SERVER_NODES.
# ## Access points # ## To operate, DAOS will need a quorum of access point nodes to be available. ## Must have the same value for all agents and servers in a system. ## Immutable after reformat. ## Hosts can be specified with or without port, default port below ## assumed if not specified. # ## default: hostname of this node access_points: ['node-4']
Define the server port by uncommenting the following line
# ## Default port # ## Port number to bind daos_server to, this will also ## be used when connecting to access points unless a port is specified. # ## default: 10001 port: 10001
Define the transport configuration (certificate files) by uncommenting the following lines
# ## Transport Credentials Specifying certificates to secure communications # transport_config: # In order to disable transport security, uncomment and set allow_insecure # to true. Not recommended for production configurations. allow_insecure: false # Location where daos_server will look for Client certificates client_cert_dir: /etc/daos/certs/clients # Custom CA Root certificate for generated certs ca_cert: /etc/daos/certs/daosCA.crt # Server certificate for use in TLS handshakes cert: /etc/daos/certs/server.crt # Key portion of Server Certificate key: /etc/daos/certs/server.key
Define the provider by uncommenting the following line. Specify a provider that is supported by your configuration.
# ## Use specific OFI provider # ## Force a specific provider to be used by all the engines. ## The default provider depends on the interfaces that will be auto-detected: ## ofi+verbs;ofi_rxm for Infiniband/RoCE and ## ofi+socket for non-RDMA-capable Ethernet. # provider: ofi+verbs;ofi_rxm
Define the socket directory by uncommenting the following line
# ## Use the given directory for creating unix domain sockets # ## DAOS Agent and DAOS Server both use unix domain sockets for communication ## with other system components. This setting is the base location to place ## the sockets in. # ## default: /var/run/daos_server socket_dir: /var/run/daos_server
Define the number of hugepages to allocate for use with the NVMe by uncommenting the following line
# ## Number of hugepages to allocate for use by NVMe SSDs # ## Specifies the number (not size) of hugepages to allocate for use by NVMe ## through SPDK. This indicates the number to be used for each spawned ## I/O Engines, so the total will be this number * number of I/O Engines. ## Default system hugepage size will be used. # ## default: 4096 nr_hugepages: 4096
Define the control plane log mask by uncommenting the following line
# ## Force specific debug mask for daos_server (control plane). ## By default, just use the default debug mask used by daos_server. ## Mask specifies minimum level of message significance to pass to logger. ## Currently supported values are DEBUG and ERROR. # ## default: DEBUG control_log_mask: DEBUG
Define the control plane log file by uncommenting the following line
# ## Force specific path for daos_server (control plane) logs. # ## default: print to stderr control_log_file: /tmp/daos_server.log
Define the privileged helper log file by uncommenting the following line
# ## Enable daos_admin (privileged helper) logging. # ## default: disabled (errors only to control plane log) helper_log_file: /tmp/daos_admin.log
Define a single DAOS engine by uncommenting the following lines
# ## When per-engine definitions exist, auto-allocation of resources is not ## performed. Without per-engine definitions, node resources will ## automatically be assigned to engines based on NUMA ratings, there will ## be a one-to-one relationship between engines and sockets. # engines: -
Define the number of I/O service threads for this engine by uncommenting the following line
# # # Targets represent the number of I/O service threads (and network endpoints) # # to be allocated per engine. # # Immutable after reformat. # # targets: 16 targets: 8
Define the number of additional offload service threads for this engine by uncommenting the following line
# # # Number of additional offload service threads to accelerate checksum # # and server-side RPC dispatch per engine. # # Immutable after reformat. # # nr_xs_helpers: 6 nr_xs_helpers: 16
Define the OFI interace and port for this engine by uncommenting the following lines
# # # Use specific OFI interfaces. # # Specify the fabric network interface that will be used by this engine. # # Optionally specify the fabric network interface port that will be used # # by this engine but please only if you have a specific need, this will # # normally be chosen automatically. # fabric_iface: ib0 fabric_iface_port: 31416
Define the log mask for this engine by uncommenting the following line
# # # Force specific debug mask (D_LOG_MASK) at start up time. # # By default, just use the default debug mask used by DAOS. # # Mask specifies minimum level of message significance to pass to logger. # # # default: ERR log_mask: INFO
Define the log file for this engine by uncommenting the following line
# # # Force specific path for DAOS debug logs (D_LOG_FILE). # # # default: /tmp/daos.log log_file: /tmp/daos_engine_0.log
Define the environment variables for this engine by uncommenting the following line
# # # Pass specific environment variables to the engine process. # # Empty by default. Values should be supplied without encapsulating quotes. # env_vars: - CRT_TIMEOUT=30Define the storage class memory (SCM) mount point for this engine by uncommenting the following line
# # # Define a pre-configured mountpoint for storage class memory to be used # # by this engine. # # Path should be unique to engine instance (can use different subdirs). # # Either the specified directory or its parent must be a mount point. # scm_mount: /mnt/daos0
Define the SCM type for this engine by uncommenting the following line.
Following changes are needed if using 'ram' as scm_class:
# # Backend SCM device type. Either use PMem (Intel(R) Optane(TM) persistent # # memory) modules configured in interleaved mode or emulate SCM with a # # tmpfs running in RAM. # # Options are: # # - "dcpm" for real SCM (preferred option), scm_size ignored # # - "ram" to emulate SCM with memory, scm_list ignored # # Immutable after reformat. # scm_class: ram # # # When scm_class is set to ram, tmpfs will be used to emulate SCM. # # The size of ram is specified by scm_size in GB units. scm_size: 16
If using 'dcpm' as scm_class make below mentioned changes:
# # Backend SCM device type. Either use PMem (Intel(R) Optane(TM) persistent # # memory) modules configured in interleaved mode or emulate SCM with a # # tmpfs running in RAM. # # Options are: # # - "dcpm" for real SCM (preferred option), scm_size is ignored # # - "ram" to emulate SCM with memory, scm_list is ignored # # Immutable after reformat. # scm_class: dcpm # # # When scm_class is set to dcpm, scm_list is the list of device paths for # # PMem namespaces (currently only one per engine supported). scm_list: [/dev/pmem0]
Define the backend block device type for this engine by uncommenting the following line
# # # Backend block device type. Force a SPDK driver to be used by this engine # # instance. # # Options are: # # - "nvme" for NVMe SSDs (preferred option), bdev_{size,number} ignored # # - "malloc" to emulate a NVMe SSD with memory, bdev_list ignored # # - "file" to emulate a NVMe SSD with a regular file, bdev_number ignored # # - "kdev" to use a kernel block device, bdev_{size,number} ignored # # Immutable after reformat. # bdev_class: nvmeDefine the backend block device configuration for this engine by uncommenting the following line. This is where the NVMe addresses obtained in step #1 are applied.
# # # Backend block device configuration to be used by this engine instance. # # When bdev_class is set to nvme, bdev_list is the list of unique NVMe IDs # # that should be different across different engine instance. # # Immutable after reformat. bdev_list: ["0000:81:00.0"] # generate regular nvme.conf # # If VMD-enabled NVMe SSDs are used, the bdev_list should consist of the VMD # # PCIe addresses, and not the BDF format transport IDs of the backing NVMe SSDs # # behind the VMD address. Also, 'disable_vmd' needs to be set to false. # bdev_list: ["0000:5d:05.5"]
Proceed to enable second engine per host only if you have enough resources, or else jump to pointer "ag"
Enable second engine.
# uncomment '-' -
Same as mentioned for first engine, define number of targets for second engine
# # Targets represent the number of I/O service threads (and network endpoints) # # to be allocated per engine. # # Immutable after reformat. # targets: 8
Define the number of additional offload service threads for second engine by uncommenting the following line
# # Number of additional offload service threads to accelerate checksum # # and server-side RPC dispatch per engine. # # Immutable after reformat. # # nr_xs_helpers: 6 nr_xs_helpers: 16
Define the OFI interace and port for second engine by uncommenting the following lines
# # # Use specific OFI interfaces. # # Specify the fabric network interface that will be used by this engine. # # Optionally specify the fabric network interface port that will be used # # by this engine but please only if you have a specific need, this will # # normally be chosen automatically. # fabric_iface: ib1 fabric_iface_port: 31516
Define the log mask for second engine by uncommenting the following line
# # # Force specific debug mask (D_LOG_MASK) at start up time. # # By default, just use the default debug mask used by DAOS. # # Mask specifies minimum level of message significance to pass to logger. # # # default: ERR log_mask: INFO
Define the log file for second engine by uncommenting the following line
# # # Force specific path for DAOS debug logs (D_LOG_FILE). # # # default: /tmp/daos.log log_file: /tmp/daos_engine_1.log
Define the environment variables for second engine by uncommenting the following line
# # # Pass specific environment variables to the engine process. # # Empty by default. Values should be supplied without encapsulating quotes. # env_vars: - CRT_TIMEOUT=30Define the storage class memory (SCM) mount point for second engine by uncommenting the following line
# # # Define a pre-configured mountpoint for storage class memory to be used # # by this engine. # # Path should be unique to engine instance (can use different subdirs). # # Either the specified directory or its parent must be a mount point. # scm_mount: /mnt/daos1
- Define the SCM type for second engine by uncommenting the following line.
Following changes are needed if using 'ram' as scm_class:
# # Backend SCM device type. Either use PMem (Intel(R) Optane(TM) persistent # # memory) modules configured in interleaved mode or emulate SCM with a # # tmpfs running in RAM. # # Options are: # # - "dcpm" for real SCM (preferred option), scm_size ignored # # - "ram" to emulate SCM with memory, scm_list ignored # # Immutable after reformat. # scm_class: ram # # # When scm_class is set to ram, tmpfs will be used to emulate SCM. # # The size of ram is specified by scm_size in GB units. scm_size: 16
If using 'dcpm' as scm_class make below mentioned changes:
# # Backend SCM device type. Either use PMem (Intel(R) Optane(TM) persistent # # memory) modules configured in interleaved mode or emulate SCM with a # # tmpfs running in RAM. # # Options are: # # - "dcpm" for real SCM (preferred option), scm_size is ignored # # - "ram" to emulate SCM with memory, scm_list is ignored # # Immutable after reformat. # scm_class: dcpm # # # When scm_class is set to dcpm, scm_list is the list of device paths for # # PMem namespaces (currently only one per engine supported). scm_list: [/dev/pmem1]
Define the backend block device type for second engine by uncommenting the following line
# # If Volume Management Devices (VMD) are to be used, then the disable_vmd # # flag needs to be set to false (default). The bdev_class will remain the # # default "nvme" type, and bdev_list will include the VMD addresses. bdev_class: nvme
Define the backend block device configuration for second engine by uncommenting the following line. This is where the NVMe addresses obtained in step #1 are applied.
bdev_list: ["0000:83:00.0"]
Copy modified server yaml file to all the server nodes at `/etc/daos/daos_server.yml`
## DAOS server configuration file. # ## Location of this configuration file is determined by first checking for the ## path specified through the -o option of the daos_server command line. ## Otherwise, /etc/daos/daos_server.yml is used. # # ## Name associated with the DAOS system. ## Immutable after reformat. # ## NOTE: Changing the name is not supported in DAOS 1.0, it must not be changed ## from the default "daos_server". # ## default: daos_server name: daos_server # # ## Access points # ## To operate, DAOS will need a quorum of access point nodes to be available. ## Must have the same value for all agents and servers in a system. ## Immutable after reformat. ## Hosts can be specified with or without port, default port below ## assumed if not specified. # ## default: hostname of this node access_points: ['node-3'] # # ## Default port # ## Port number to bind daos_server to, this will also ## be used when connecting to access points unless a port is specified. # ## default: 10001 port: 10001 # ## Transport Credentials Specifying certificates to secure communications # transport_config: # # In order to disable transport security, uncomment and set allow_insecure # # to true. Not recommended for production configurations. allow_insecure: false # # # Location where daos_server will look for Client certificates client_cert_dir: /etc/daos/certs/clients # # Custom CA Root certificate for generated certs ca_cert: /etc/daos/certs/daosCA.crt # # Server certificate for use in TLS handshakes cert: /etc/daos/certs/server.crt # # Key portion of Server Certificate key: /etc/daos/certs/server.key # # ## Fault domain path # ## Immutable after reformat. # ## default: /hostname for a local configuration w/o fault domain #fault_path: /vcdu0/rack1/hostname # # ## Fault domain callback # ## Path to executable which will return fault domain string. ## Immutable after reformat. # #fault_cb: ./.daos/fd_callback # # ## Use specific OFI provider # ## Force a specific provider to be used by all the engines. ## The default provider depends on the interfaces that will be auto-detected: ## ofi+psm2 for Omni-Path, ofi+verbs;ofi_rxm for Infiniband/RoCE and finally ## ofi+socket for non-RDMA-capable Ethernet. # provider: ofi+verbs;ofi_rxm # # ## global CRT_CTX_SHARE_ADDR shared with client #crt_ctx_share_addr: 1 # # ## global CRT_TIMEOUT shared with client #crt_timeout: 30 # # ## NVMe SSD whitelist # ## Only use NVMe controllers with specific PCI addresses. ## Immutable after reformat, colons replaced by dots in PCI identifiers. ## By default, DAOS will use all the NVMe-capable SSDs that don't have active ## mount points. # #bdev_include: ["0000:81:00.1","0000:81:00.2","0000:81:00.3"] # # ## NVMe SSD blacklist # ## Only use NVMe controllers with specific PCI addresses. Overrides drives ## listed in bdev_include and forces auto-detection to skip those drives. ## Immutable after reformat, colons replaced by dots in PCI identifiers. # #bdev_exclude: ["0000:81:00.1"] # # ## Disable VFIO Driver # ## In some circumstances it may be preferable to force SPDK to use the UIO ## driver for NVMe device access even though an IOMMU is available. Note ## that use of the UIO driver requires that DAOS must run as root. # ## default: false #disable_vfio: true # # ## Disable VMD Usage # ## In some circumstances it may be preferable to not use Intel Volume Management ## Devices even if available. VMD needs to be available and configured in the ## system BIOS before use. # ## default: true #disable_vmd: false # # ## Use Hyperthreads # ## When Hyperthreading is enabled and supported on the system, this parameter ## defines whether the DAOS service thread should only be bound to different ## physical cores (value 0) or hyperthreads (value 1). # ## default: false #hyperthreads: true # # ## Use the given directory for creating unix domain sockets # ## DAOS Agent and DAOS Server both use unix domain sockets for communication ## with other system components. This setting is the base location to place ## the sockets in. # ## default: /var/run/daos_server socket_dir: /var/run/daos_server # # ## Number of hugepages to allocate for use by NVMe SSDs # ## Specifies the number (not size) of hugepages to allocate for use by NVMe ## through SPDK. This indicates the number to be used for each spawned ## I/O Engines, so the total will be this number * number of I/O Engines. ## Default system hugepage size will be used. # ## default: 4096 nr_hugepages: 4096 # # ## Force specific debug mask for daos_server (control plane). ## By default, just use the default debug mask used by daos_server. ## Mask specifies minimum level of message significance to pass to logger. ## Currently supported values are DEBUG and ERROR. # ## default: DEBUG control_log_mask: DEBUG # # ## Force specific path for daos_server (control plane) logs. # ## default: print to stderr control_log_file: /tmp/daos_server.log # # ## Enable daos_admin (privileged helper) logging. # ## default: disabled (errors only to control plane log) helper_log_file: /tmp/daos_admin.log # # ## Enable daos_firmware (privileged helper) logging. # ## default: disabled (errors only to control plane log) #firmware_helper_log_file: /tmp/daos_firmware.log # # ## When per-engine definitions exist, auto-allocation of resources is not ## performed. Without per-engine definitions, node resources will ## automatically be assigned to engines based on NUMA ratings, there will ## be a one-to-one relationship between engines and sockets. # engines: - # # Rank to be assigned as identifier for this engine. # # Immutable after reformat. # # Optional parameter, will be auto generated if not supplied. # # rank: 0 # # # Targets represent the number of I/O service threads (and network endpoints) # # to be allocated per engine. # # Immutable after reformat. # targets: 8 # # # Pin this engine instance to cores and memory that are related to the # # NUMA node ID specified with this value. For best performance, it is # # necessary to select a NUMA node that matches that of the fabric_iface. # # Optional parameter, set either this option or first_core but not both. # # pinned_numa_node: 0 # # # Offset of the first core to be used for I/O service threads. # # Immutable after reformat. # # Optional parameter, set either this option or pinned_numa_node but # # not both. # # first_core: 0 # # # Number of additional offload service threads to accelerate checksum # # and server-side RPC dispatch per engine. # # Immutable after reformat. # nr_xs_helpers: 16 # # # Use specific OFI interfaces. # # Specify the fabric network interface that will be used by this engine. # # Optionally specify the fabric network interface port that will be used # # by this engine but please only if you have a specific need, this will # # normally be chosen automatically. # fabric_iface: ib0 fabric_iface_port: 31416 # # # Force specific debug mask (D_LOG_MASK) at start up time. # # By default, just use the default debug mask used by DAOS. # # Mask specifies minimum level of message significance to pass to logger. # # # default: ERR log_mask: INFO # # # Force specific path for DAOS debug logs (D_LOG_FILE). # # # default: /tmp/daos.log log_file: /tmp/daos_engine_0.log # # # Pass specific environment variables to the engine process. # # Empty by default. Values should be supplied without encapsulating quotes. # env_vars: - CRT_TIMEOUT=30 # # # Define a pre-configured mountpoint for storage class memory to be used # # by this engine. # # Path should be unique to engine instance (can use different subdirs). # # Either the specified directory or its parent must be a mount point. # scm_mount: /mnt/daos_0 # # # Backend SCM device type. Either use PMem (Intel(R) Optane(TM) persistent # # memory) modules configured in interleaved mode or emulate SCM with a # # tmpfs running in RAM. # # Options are: # # - "dcpm" for real SCM (preferred option), scm_size ignored # # - "ram" to emulate SCM with memory, scm_list ignored # # Immutable after reformat. # scm_class: dcpm scm_list: [/dev/pmem0] # # # When scm_class is set to ram, tmpfs will be used to emulate SCM. # # The size of ram is specified by scm_size in GB units. # scm_size: 16 # # # Backend block device type. Force a SPDK driver to be used by this engine # # instance. # # Options are: # # - "nvme" for NVMe SSDs (preferred option), bdev_{size,number} ignored # # - "malloc" to emulate a NVMe SSD with memory, bdev_list ignored # # - "file" to emulate a NVMe SSD with a regular file, bdev_number ignored # # - "kdev" to use a kernel block device, bdev_{size,number} ignored # # Immutable after reformat. # bdev_class: nvme # # # Backend block device configuration to be used by this engine instance. # # When bdev_class is set to nvme, bdev_list is the list of unique NVMe IDs # # that should be different across different engine instance. # # Immutable after reformat. bdev_list: ["0000:81:00.0"] # generate regular nvme.conf # # If VMD-enabled NVMe SSDs are used, the bdev_list should consist of the VMD # # PCIe addresses, and not the BDF format transport IDs of the backing NVMe SSDs # # behind the VMD address. Also, 'disable_vmd' needs to be set to false. # bdev_list: ["0000:5d:05.5"] - # # Rank to be assigned as identifier for this engine. # # Immutable after reformat. # # Optional parameter, will be auto generated if not supplied. # # rank: 1 # # # Targets represent the number of I/O service threads (and network endpoints) # # to be allocated per engine. # # Immutable after reformat. # targets: 8 # # # Pin this engine instance to cores and memory that are related to the # # NUMA node ID specified with this value. For best performance, it is # # necessary to select a NUMA node that matches that of the fabric_iface. # # Optional parameter, set either this option or first_core but not both. # # pinned_numa_node: 1 # # # Offset of the first core to be used for I/O service threads. # # Immutable after reformat. # # Optional parameter, set either this option or pinned_numa_node but # # not both. # # first_core: 22 # # # Number of additional offload service threads to accelerate checksum # # and server-side RPC dispatch per engine. # # Immutable after reformat. # nr_xs_helpers: 16 # # # Use specific OFI interfaces. # # Specify the fabric network interface that will be used by this engine. # # Optionally specify the fabric network interface port that will be used # # by this engine but please only if you have a specific need, this will # # normally be chosen automatically. # fabric_iface: ib1 fabric_iface_port: 31516 # # # Force specific debug mask (D_LOG_MASK) at start up time. # # By default, just use the default debug mask used by DAOS. # # Mask specifies minimum level of message significance to pass to logger. # # # default: ERR log_mask: INFO # # # Force specific path for DAOS debug logs. # # # default: /tmp/daos.log log_file: /tmp/daos_engine_1.log # # # Pass specific environment variables to the engine process. # # Empty by default. Values should be supplied without encapsulating quotes. # env_vars: - CRT_TIMEOUT=100 # # # Define a pre-configured mountpoint for storage class memory to be used # # by this engine. # # Path should be unique to engine instance (can use different subdirs). # scm_mount: /mnt/daos_1 # # # Backend SCM device type. Either use PMem (Intel(R) Optane(TM) persistent # # memory) modules configured in interleaved mode or emulate SCM with a # # tmpfs running in RAM. # # Options are: # # - "dcpm" for real SCM (preferred option), scm_size is ignored # # - "ram" to emulate SCM with memory, scm_list is ignored # # Immutable after reformat. # scm_class: dcpm # # # When scm_class is set to dcpm, scm_list is the list of device paths for # # PMem namespaces (currently only one per engine supported). scm_list: [/dev/pmem1] # # # Backend block device type. Force a SPDK driver to be used by this engine # # instance. # # Options are: # # - "nvme" for NVMe SSDs (preferred option), bdev_{size,number} ignored # # - "malloc" to emulate a NVMe SSD with memory, bdev_list ignored # # - "file" to emulate a NVMe SSD with a regular file, bdev_number ignored # # - "kdev" to use a kernel block device, bdev_{size,number} ignored # # Immutable after reformat. # # # When bdev_class is set to malloc, bdev_number is the number of devices # # to allocate and bdev_size is the size in GB of each LUN/device. # bdev_class: malloc # bdev_number: 1 # bdev_size: 4 # # # When bdev_class is set to file, Linux AIO will be used to emulate NVMe. # # The size of file that will be created is specified by bdev_size in GB units. # # The location of the files that will be created is specified in bdev_list. # bdev_class: file # bdev_list: [/tmp/daos-bdev1,/tmp/daos-bdev2] # bdev_size: 16 # # # When bdev_class is set to kdev, bdev_list is the list of unique kernel # # block devices that should be different across different engine instance. # bdev_class: kdev # bdev_list: [/dev/sdc,/dev/sdd] # # # If Volume Management Devices (VMD) are to be used, then the disable_vmd # # flag needs to be set to false (default). The bdev_class will remain the # # default "nvme" type, and bdev_list will include the VMD addresses. bdev_class: nvme bdev_list: ["0000:83:00.0"]